5. EXPERIMENTAL DESIGN

One of the main goals of the present research is to investigate the effectiveness of the Automated Debugger for Ada Tasks (ADAT) as a tool for teaching concurrency. This chapter describes the research methodology employed in the investigation. The setting, the subjects, design procedures of the research, the instruments used to collect data, and the pilot research are discussed.

The null hypothesis asserts that no difference exists between the two groups in the first stage. One group will use ADAT and the other will not.

b) The use of the ADAT provides an improvement in the understanding of concurrency.

The null hypothesis asserts that no difference exists between the two groups in the second stage. Literally, what is being tested is whether or not the use of ADAT facilitated transfer of concepts.The independent variable is the use of ADAT and its omission. The main dependant variables will be the time spent working on each program and whether each program was corrected (or extended) or not. Complementary dependent variables may also be selected from the data collected from the demographic survey (Appendix A) and the data collected from the films which are shown in Appendix G.

The subjects of this study consisted of 40 undergraduate and graduate students enrolled in the second year course Programming and Data Structures and in the third year course Theory of Computer Translators, offered by the Electrical Engineering and Computer Science Department at The George Washington University.

The experiment was performed using two groups of subjects, the experimental group (E) and the control group (C), in two stages. Each group contained twenty subjects.

In stage one, the subjects in the E group used the Small-Ada system with the ADAT tool available, and the subjects in the C group used the Small-Ada system without the ADAT tool. Subjects from both groups were asked to find and correct one bug in each of two Small-Ada programs. The SAPM was available in both groups.

Based on several semesters of teaching concurrency, a set of typical conceptual errors made by novice programmers were noted. The work of Moran [Mora85] provided additional information. These errors were typified in the first three programs, named Pgm0, Pgm1, and Pgm2 (see Appendix C).

In Stage Two, both groups were asked to use the Small-Ada system to extend a Small-Ada program. At this stage of this experiment, the conditions under which the subjects worked were identical. The program used in this stage was named Pgm3 (see Appendix C).

Four programs were used. The first program, Pgm0, was used only for warm-up purposes. Pgm0 has three explicit tasks. Each task outputs the contents of a specific global variable. The error in this program is that the global variables used by the tasks have their declarations elaborated concurrently with the execution of each task. The result is that some of the tasks may output incorrect values. This problem falls into the initialization anomaly category [Lope91c].

Two programs (Pgm1 and Pgm2) were used in the first stage and one program (Pgm3) was used in the second stage. Pgm1 resembles the situation of a teacher working at his or her office with three students waiting outside the office to check their grades. Pgm1 has five tasks: the main Small-Ada program, the teacher task and the three student tasks. Each student task issues an entry call to the teacher task, asking for his or her grade. After the call has been completed, it terminates. This is a typical problem of a server (the teacher) and clients (the students). The problem with Pgm1 is that the teacher task will only service the first entry call and then it will terminate. What is missing in this program is a complete server implementation able to service students until all of them are terminated. This problem falls into the server anomaly category.

The second program used in Stage One is Pgm2. Pgm2 has three tasks, the main Small-Ada program and two explicit tasks. The main program outputs the contents of two global variables that have their values assigned within the two tasks. In this program, each task is making use of global variables to communicate with the main program. The problem with Pgm2 is that it allows race conditions to occur. Pgm2 does not contain any synchronization statements to assure that read operations on global variables are not carried out concurrently with update operations within different tasks. Consequently, the output of the program does not reflect the work performed by each of the tasks. This situation is referred to in the literature as a shared data problem that results from race conditions. This form of task sequencing error is referred to here as an update anomaly. The results obtained in Stage One are used to test the null hypothesis of no difference between the two groups.

Pgm3 was used in Stage Two. This program does not contain any bugs. It presents a solution to the problem of finding the maximum value stored within a vector. Two identical tasks are used to find the maximum value of each half of the vector. After being elaborated, each task waits (by means of an accept statement) for both the lower and upper bounds of the vector which specify the interval of work for a task. After this synchronization point is completed, the two explicit declared tasks run asynchronously until the maximum value is found. The main program is waiting for the maximum value found by each task (passed to it by an out parameter by means of an entry call issued by the main program to each task). After these final two entry calls have been completed, the main program outputs the maximum value between the two values received from the two tasks. As mentioned before, this program does not contain any bugs. The subjects were asked to extend the program so that it will find the maximum value by using four tasks instead of two. The results obtained in Stage Two are used to test the null hypothesis of no difference between the two groups. Here ADAT is being tested to determine if its use improves the understanding of concurrency by novice concurrent programmers. That is, do the use of ADAT show transfer when used by novice concurrent programmers.

Students from both groups received identical lectures on the Ada tasking model. The subjects did not receive any training on SAPM, SAIE or ADAT.

The procedure used in this investigation involved a number of steps which are described below:

A lecture and reading material covering concurrency and the syntax of the Ada tasking model were given to the subjects..

The subjects were asked to complete a demographic data sheet, shown in Appendix A.

The demographic data sheet (see Appendix A) was used to sort the subjects by their GPA. Then, starting with the top of the sorted list, subjects were randomly assigned to each of the groups. The two programs used in the first stage were randomly assigned to each subject.

The subjects read the directions sheet, shown in Appendix B, then performed the experiment. They were given a quick reference manual on Small-Ada together with a hard copy for each of the four programs (Pgm0, Pgm1, Pgm2, and Pgm3).

A warm-up faulty program (Pgm0) was given to the subjects to work with for up to 10 minutes with the assistance of the author.

The subjects then worked with the remaining two faulty Ada programs (Pgm1 and Pgm2), which are shown in Appendix C. The subjects of the E Group used the Small-Ada system with the ADAT tool in the first stage, while the subjects of the C Group used Small-Ada without the ADAT tool in the first stage.

In the second stage, all subjects received the same treatment. That is, they all used Small-Ada (with SAPM and without ADAT) and were asked to extend a concurrent program (Pgm3).

After working with programs Pgm1, Pgm2 and Pgm3, the subjects then completed the follow up questionnaire, which is shown in Appendix E.

Each subject was videotaped during the experiment. The tape of each subject was used later to collect the data that is shown in Appendices D and G. Each subject was left alone in a room with a VHS camera running. The subjects were instructed to spend no more than 20 minutes working on each program.

Pilot research was conducted in April of 1991 [Lope91d] to determine the suitability of the instruments, to refine them, and to obtain experience with the logistics of the research. Fifteen students in the Electrical Engineering and Computer Science Program who were enrolled in a junior Introduction to Operating Systems class participated.

The experiment was explained to the students early in the semester ; a lecture on the Ada tasking model was given later in the semester. The experimental sessions were then scheduled. Three programs taken from Moran's work [Mora85] were used. Each program contained one bug. The first program was used as a warm-up program. Three groups were used. Group One used only paper to debug. Group Two used the standard system. Group Three used the SAPM. No significant statistical difference was found in the experiment, due to the small number of subjects used per group: six in Group One, four in Group Two, and five in Group Three. Table 5.1 summarizes the results obtained from the pilot experiment.

GROUP	FIXED/NOT FIXED	AVG. TIME (min)
One	2/5	19
Two	4/0	16
Three	5/0	12

The results obtained from the pilot study provided the justification to proceed with the actual experiment. The next chapter will discuss the experimental results which show how novice concurrent programmers reacted to the use of ADAT.