Introduction

The proliferation of web-based delivery systems in higher education has generated a great deal of speculation about the kind of impact these products will have on teaching and learning. Designed around course management features, courseware applications such as WebCT and Blackboard offer faculty a relatively fast and easy method for creating a medium that mimics the activities of the conventional classroom.

Using a web browser to access the courseware, students can participate in bulletin boards, chat rooms, and web-publishing projects in an environment in which every page attempts to project a uniform "look and feel". In turn, faculty can track individual students' progress through "pages" of text by way of the course management database (a surveillence feature to which some students strenuously object). Being web-based means that all of this is done remotely — student and teacher are not confined by the time-space limitations of a scheduled classroom.

Much has been written about the flexibility web-based courses offer students in the pacing and sequencing by which they proceed through course content (Downs, Carlson, Repman & Clark, 1999). But very little is known about the impact of web-based instruction as it relates to the instructor's flexibility in teaching content.

One of the primary concerns that technology-adept faculty have about prepackaged courseware such as WebCT is the apparent constraints it places on their ability to adapt and improvise as they become more comfortable with web-based tools. Most users are content just to be able operate the WebCT environment without having to know HTML, CGI scripts, Java and a variety of other web programming languages that perplex and annoy our right-brain sensibilities. A relatively small number of faculty often characterized as "early adopters" can eventually see past the simple, prescribed applications of the tools in WebCT and Blackboard and wish to do more specialized kinds of teaching by capitalizing on the hidden assets of technology (Jacobsen,1998).

Instructional Cycles and Conditional Release

The general perception that web-based courses capture only the most superficial features of the traditional classroom experience has not deterred some of these more enterprising faculty who have found ways to work around the apparent limitations of the courseware used to author these courses. They have devised clever routines for using the available tools and functions in ways that may not have been anticipated by the publishers.

One particular type of innovation is based on the creation of "instructional cycles" that use WebCT's "conditional release" features, and more importantly, the statistical analysis of online (web-based) quizzes. Faculty use these cycles to harness the feedback created by student responses to online quizzes. The feedback serves as data that can be used as the basis for formative assessments of how teaching is or is not effecting student learning (Angelo and Cross,1993).

Using the electronic classroom assessment techniques like those available in WebCT's online quiz, the instructor can receive information about the effectiveness of their teaching "while simultaneously providing students with feedback on their learning, and helping to shape the instructional moment in ways not previously possible" (Gandolfo and Carver, 1995). As Gandolfo and Carver have eloquently stated, the real value of online testing is the speed with which faculty can redirect their subsequent lectures to address deficiencies in learning and teaching.

The Case Study: Undergraduate Psychology

Robert Holt is an Associate Professor of Psychology in George Mason's College of Arts and Sciences. Holt first introduced some innovations into his undergraduate course on social psychology using WebCT's online quiz tools in the 1999 Fall semester. Holt's WebCT course is notable for two reasons.

First, his use of the quiz tool allowed students to receive immediate feedback about how successfully they were learning the key concepts in their reading assignments. Using these online quiz results, Holt identified those concepts that the students had trouble grasping and made the necessary adjustments to his classroom teaching to address the problem areas. As a consequence of these online assessments, the classroom lectures were able to anticipate the needs of the students in ways that better promoted the instructor's stated learning outcomes.

In this 200-level survey course in social psychology that emphasizes theories, methods and data in the field, Holt's original instructional cycle was based solely on the twelve chapters of the course text and classroom lectures. This included an assigned reading of one chapter in the text each week, a lecture and class discussion, three exams during the semester and a final exam at the end of the semester (Figure 1). Typically, the entire instructional cycle for each chapter was covered in one week.

Figure 1: Traditional Instructional Cycle

The New Cycle: Self-Assessment

With the introduction of WebCT, the instructional cycle was redesigned to introduce new activities (Figure 2). Using the quiz tool, Holt developed a series of self-assessment tests and quizzes for each chapter. Students were required to complete the self-assessment test by logging on to the web site typically 24-48 hours prior to attending class.

Figure 2: Web-Enhanced Instructional Cycle

Each self-assessment question consisted of a topic or concept associated to a key term in the text. The number of key terms ranged from 15 to 25 across the twelve chapters. All key terms were listed for each question in the format of a very large "multiple choice" question. Functionally, however, the response to the self assessment items was more similar to matching the topic or concept with the associated key term. In this process, some key terms were associated with more than one topic or concept while other key terms did not have any associated material.

To decrease evaluation anxiety, the self-assessment had no time limit and students were given two attempts to increase their scores if they so desired. Immediate, detailed feedback on which items the student answered correctly and incorrectly was provided after each self-assessment. The self-assessment scores did not, however, directly impact the students' cumulative total scores for the class. Rather, students were given five points for completing the self-assessment for a chapter and zero if they did not attempt the self-assessment at all. Thus, the grade points for self-assessments ranged from a possible minimum of 0 to a maximum of 60 points for completing self-assessments for all 12 chapters.

As an additional incentive for completing the self-assessment, students could gain a set of class notes in PowerPoint form after completing the self-assessment. For the first two sections of the course (the first eight chapters), the student could obtain the PowerPoint slides for the class discussions if he or she answered at least one self-assessment item correctly. With this low criterion, some students reported taking the self assessment without thoroughly reading the chapter in the hope of getting at least one correct. To counteract this tendency, for the last section of the class the criterion for obtaining the PowerPoint slides was raised to answering 25% of the self-assessment items correctly.

Text, Lecture and Quiz

Similarly, students were required to complete the quiz for a chapter by logging on to the web site after the last class related to that chapter. Students were free to take the quiz any time after the last class associated with a chapter but before classes began on the subsequent chapter. Typically this time window was 2 to 4 days.

Classes covered text content but also presented new, related material. The content for quizzes reflected both the text and class material. Text items were systematically selected from the computerized item test bank provided by the text book publisher. In addition, the instructor designed multiple choice and matching items that covered the new material for each chapter that was presented during classes.The proportion of items across all quizzes covering each type of material was approximately 2/3 to 3/4 text-focused questions with the remainder being items focused on class content.

The conditions for taking quizzes were quite distinct from those for the self-assessments. Quizzes had a time limit that was designed to be sufficiently restrictive to prevent the students' answering questions by looking up the material in the text. Students were only allowed one attempt for a quiz and had to complete it within the specified time limit. As with the self-assessments, students were given immediate, detailed feedback on correct and incorrect answers.

Quiz scores were included in the students' cumulative total scores for the class. In total, quiz scores had a sizable impact on student grades because the total possible points for all quizzes was 445 points compared to the possible 489 points for all three exams and the final exam (Figure 3). Before and after each class, Holt would access the quiz information database and check the results of the student's self-assessment tests and quizzes. He would use this data on the students' performance on the online assessment in 2 ways.

Figure 3: Grade Components

The Application of Feedack Data

When checking the students self-assessment tests results before each class Holt would capture the frequency distribution graph generated by WebCT along with the test item as a bitmap for each of the questions and display them at the beginning of each class. The graph would show the distribution of the number of students who answered the question correctly as well as the distribution of incorrect guesses made by the class. The pattern of incorrect answers for each item determined how he would use this information to facilitate discussion in the class. If all the members of the class tended to make the same incorrect guess, Holt would emphasize why that particular answer was wrong. If the distribution of wrong guesses was evenly scattered then he would emphasize why a particular answer was correct.

Holt would also annotate his class notes wherever the lecture material touched on concepts with which the students experienced problems during the online assessments. Whenever he would encounter one of the annotations as the lecture progresed, it would prompt him to spend more time explaining the concepts or terms with which the students had problems.

The Impact of WebCT Innovations on Learning Outcomes

Only one source of data suggested that the online testing and adjustments to the class lectures had a positive impact on the learning outcomes. Prior to the WebCT intervention, Holt typically had to adjust the absolute grading curve downward some 10 percentage points (60/59% to 50/49%) to avoid failing an unacceptable number of students. However for the first time in the twelve years during which he has taught the course, he was required to drop the grading curve by only 1 percentage point.

Based on the same grading method, this would indicate that the performance of this class is about 5-9% above the performance of previous classes without the WebCT intervention. This interpretation of the data must be viewed with some caution since extraneous variables such as changes in course content and student composition of the class may have had an impact on the reliability of the grading scheme.

Student evaluations of the WebCT components of the course were mixed. Student responses concerning the self-assessments were broadly categorized into positive, neutral, and negative responses. Approximately one-third of the students had positive responses, one-third were neutral, and one-third had negative responses. The negative responses primarily focused on a perceived lack of effectiveness rather than a direct negative effect on learning. The positive responses primarily focused on keeping up with the readings and learning the class material. The positive responses imply that some students perceived a direct positive effect of the self-assessments on motivation to learn.

Student responses concerning the on-line quizzes were categorized into the same positive, neutral, and negative categories (Figure 4). Classifying responses to the quizzes was more difficult because each quiz contained two sets of focal questions: a set focused on the text material and a set focused on the associated class material. Approximately one-third of the students had positive responses. However, 11 of the 28 students were neutral or ambivalent, which was a slightly higher proportion than for the self-assessments. Several students seemed to have an ambivalent overall response due to different reactions to the text and class-focused questions. A minority of five students had dominantly negative responses about the quizzes.

Figure 4: Sample Student Comments

Innovation and Risk

The student evaluations can also be analyzed to obtain some information on the risks of making teaching innovations such as the modified instructional cycle. Since student evaluations are used for tenure, promotion, and salary decisions, the impact of innovation on student evaluations may pose a risk to the instructor. This issue of risk was assessed with two types of analyses.

First, the overall rating of this course was compared to the department, college, and university norms as well as the instructor's past history of course evaluations for this course. The overage rating for this course was 4.06 (N = 28) on a 5-point scale ranging from poor (1) to excellent (5). This mean overall evaluation was compared to the means for same question for the department (4.36), the college (4.25) and the university (4.26). The overall class rating is not significantly lower than any of these reference values (t (28) = -1.56, -0.97, and -1.02, respectively, which are non-significant differences).

Since this instructor typically had course evaluations at or above the department and college means, the overall class evaluation is also slightly lower than his typical evaluation. Given the variability in the class evaluations across students and the limited sample size, the safest conclusion for the overall ratings is that the innovation did not appear to make a strong negative impact on course evaluations although it may have had a slight negative impact.

The second analysis to evaluate the risk of innovation was to compare the qualitative student evaluations of the self-assessments and on-line quizzes with their final course evaluations. If student positive or negative reactions to innovation would strongly color their evaluations of the course, this potentially poses a risk to the instructor. To examine this issue, the student positive, neutral, or negative reactions to self-assessments and on-line quizzes were used to predict each of six class evaluation items. None of the six analyses approached statistical significance.

One final analysis was to condense the six class evaluation items into one factor score representing overall class evaluation. However, the student reactions to instructional innovations also did not predict that overall composite (F (2,25) = .071, non-significant). Therefore, within the limits of sample size and precision of measurement of all variables, there is no evidence of any carry-over of student reactions to innovation to the course evaluations. Based on this evidence, there is no general negative impact of innovation on student evaluations. That is, the risk to the instructor for innovation will depend very much on precisely how the innovation is constructed and implemented.

Poorly developed or implemented innovations could certainly impact adversely on student reactions and class evaluations. Conversely, well-developed and implemented innovations could positively affect reactions and evaluations. The potential negative risk in innovation may approximate the known negative effects of teaching a course for the first time (McKeachie and Chism, 1994) as compared to subsequent iterations of the class.

Advantages and Disadvantages of Technology-Enriched Innovation

Holt perceived a variety of advantages and disadvantages to the integration of WebCT in the course. Some of the disadvantages included the time and effort to design and implement the web-based materials on a weekly basis. Instructors considering such innovations should carefully consider the resources available, such as research assistants or release time for course construction, before deciding to do wholesale implementations of technology.

The bottom-line advantage for the instructor was the potential increased learning due to the innovations in the instructional cycle. Holt's experience in this pilot study seem to support the argument that web-based assessment is a pedagogically sound strategy to generate feedback on student's level of competence (Chetty, 2000). Compared to a baseline of similar courses in the past, the instructor's criterion for failing grades was much higher (59% vs. a typical 50%) without incurring a large proportion of failing grades. However, this result may have as much to do with a selective attrition of the worst-performing students as it does to the presence of web-based self-assessments and quizzes.

Student responses to the innovations covered the entire spectrum of positive, neutral, and negative responses. About one-third of the students responded positively to the self-assessments and another one-third to the on-line quizzes. Students seemed to react differently to these two major components of the changed instructional cycle. This indicates that the exact nature of the components of an innovation may react with individual differences among students in determining their reactions. Delineating the key individual differences that moderate a student's reaction to different components may be a focus of future research.

Conclusion

Web technology offers a diverse set of possible innovations to the traditional teaching process. This paper summarizes one such effort to augment conventional in-class teaching with web-based quizzes before and after each assigned chapter. Bob Holt's motivations for implementing the online quizzes were two-fold. Rather than spend precious class time introducing basic definitions and simple principles, Holt used the quizzes to compelled his students to "open [and read] the textbook before each class." This increased the prospects of using class time for elaborating on the more interesting portions of the text. Receiving fast, systematic analysis and feedback was clearly the foremost reward for the "treadmill of time-costly activities" required to manage the twice-weekly assessments. The opportunity to quickly adapt to the needs of his students gave Holt a real sense that he "was targeting [his] teaching efforts more effectively."

Over the years, many technological innovations in education have raised expectations only to later disappoint. In the final analysis, they failed because they were unable to add real value to the pedagogy (Fincher, 1998). At its best, instructional technology should be able to transform student complacency and give us a better view of the learning process. The current staples of digital teaching, e-mail and bulletinboards, can provide continuous feedback and assessment. They offer us a feel for the quality of an individual student's work. The statistical data derived from web-based quizzes, on the other hand, offers a snapshot detailing the progress of an entire class at any desired time.

Students are better learners and learn more deeply when they receive timely, specific feedback. The same axiom applies to teaching. The tools now available through the web offer new opportunities for inquiry and discovery about the instructional cycle. For student and teacher, it is an invitation to reflect, to reexamine our assumptions, and to build cooperation for improved learning.

Notes and References

Angelo, T. A., & Cross, P. C. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. (2nd ed.). San Franscisco: Jossey-Bass.

Chetty, M. (2000). A scheme for on-line Web-based assessment. Engineering Science and Education Journal, 9, (1) 27-32.

Downs, E., Carlson, R. D., Repman, J., Clark, K. (1999, March). Web-Based Instruction: Focus on Learning. Paper presented at the Society for Information Technology & Teacher Education International Conference, San Antonio, TX.

Fincher, C. (1998, December). Teaching and Technological Innovation. IHE Newsletter. Athens, GA: Institute of Higher Education, University of Georgia, 3-8.

Gandolfo, A. & Carver, C. A., (1995). Electronic Classroom Assessment Techniques: Assessment Beyond the Classroom in a Networked Environment. Assessment Update, 7, (6) 3.

Jacobsen, M., (1998). Adoption Patterns of Faculty Who Integrate Computer Technology for Teaching and Learning in Higher Education. Paper presented at the Association for the Advancement of Computering in Education's World Conference of Educational Media, Hypermedia & Telecommunications, Frieberg, Germany.

McKeachie, W.J.,& Chism, N., (1994). Teaching tips: strategies, research, and theory for college and university teachers. (9th ed.). Lexington, MA : D.C. Heath

Jerry Drake received his M.Ed. in instructional technology from George Mason University in 1995 and is currently a doctoral student at Mason’s Graduate School of Education. He is employed as a faculty support analyst by the Instructional Resource Center at George Mason where he advises faculty on the application of instructional technologies to teaching. In a career that spans 26 years, Drake is also a writer and award-winning documentary filmmaker.