Evaluation Instruments and Models for Distance Education Materials
Criteria for the Evaluation of Distance Learning Materials
The purpose of evaluation is to find out the extent to which the goals or objectives of an educational activity are being achieved. Reiser and Gagne state that selection of media is a "burning" question in order to make instruction optimally effective (1983, p. 3) and they observe that much instruction is not planned to be optimally effective.
Existing media selection models variously emphasize physical features or human senses. Clark and Angert (1981) reviewed media selection models and concluded that they are preoccupied with technical considerations such as convenience and portability and are weak on instructional design considerations. Schramm (1977) points out that no procedure can be applied to all situations and guidelines should consider local needs, situations and resources. Bates (1980) states that the primary concern is how the media interact. A literature review produced the following concerns.
Instrument Terminology
When the term understanding or appreciation is used, it should delineate the specific nature by student behaviors.
What values are highlighted? The program is in keeping with the principles that guide the user institution. Materials represent artistic, historic and literary qualities.
How are educational objectives selected? The student is central to the learning experience; evaluation should be done within the total context of student learning; educational needs are defined so that they can be met for the educational system and individual programs; expected changes in student behavior, attitudes or interest are defined; curricular objectives are stated; media contributes to specific instruction goal achievement; the extent to which stated objectives are achieved; objectives are stated by cognitive, affective and psychomotor domain; objectives are measurable and can measure success or failure; lesson objectives give adequate direction for student study; and whether students can correctly identify educational objectives.
Characteristics of students should be known including their initial competence in the topic. Material should be suitable for learners with an appropriate level of content complexity and vocabulary which accommodates ability differentials.
Compare the similarity of the campus class with the electronically mediated class; objectives, course experiences and content should be equivalent. Supplements or experiences can be developed or adapted to make the courses similar; the course should be adaptable to many teaching situations, populations and methods; and the course should be of interest to students as a required, elective, or interdisciplinary course. A report, such as a program producer's field evaluation of student learning, should be available to provide learner verification data on the product's effectiveness. The method for evaluation and assessment which has been validated should be described and the evaluation should be directly related to the course objectives.
The delivery method should be considered: loaned tapes are available when needed, facilitate repetition, search and mastery, analysis, relating and reflection, are easier to integrate. Broadcast programs are shown perhaps once at a fixed time, do not facilitate repetition, search, mastery, analysis, relating or reflection and are more difficult to integrate. The control characteristics of cassettes should be exploited, of segment use, clear stopping points, use of activities, indexing, close integration with other media (text, etc.) and concentration on audio-visual aspects so that the video cassette is to the broadcast what the book is to the lecture.
Instructional DesignConsider the schedule of learning set up for the student so that students are not overloaded. Consider the time required to complete the course; the number of lessons; appropriate segment length stated instructional objectives; it is fully planned and has an appropriate level of abstraction; uses visual, audio and tactile components; directs student activity toward specified learning outcomes by frequent overt and covert responses; the familiar is used as a bridge to the unfamiliar; and a range of direct and indirect methods is used. The material should be broken into manageable chunks; the first two lessons are shorter; lesson size is easily managed, not too long or difficult to discourage students; lessons are self-paced to allow student planning; and the production pacing maintains interest. Telecourse components should be examined for high quality; components should make learning experiences occur; accomplish individual objectives for which they were created; utility of each component part; provide realia (real objects); effectively use graphics; components should be easy to use; useful; well packaged; transportable; available; have an appropriate quantity; should include concepts of appropriate difficulty; relate ideas and link discussion.
Components should be examined for relevance of reading rates, speed vs. critical reading; readability; use of unexplained technical terms; overall coherence and consistency; argumentative and indices of fallacious reasoning; does not make assumptions, draw conclusions in error, or masquerade examples as definition or opinions as fact; clarity; well phrased instructions and questions; have complete, adequate and useful proofs; show a balance of active and passive assignments; should contain self assessment questions and activities to make the student think and evaluate progress; and have appropriate role, position and function of summaries. Material should appeal to the students' interests, achievement and background; and provide a stimulus to creativity. Components should correlate well with one another so that they are integrated.
Self instruction should be encouraged through strategies which motivate student learning, hold student attention and stimulate students. Students should be provided with help to develop basic learning skills such as fast and selective reading, essay writing, development of objectivity and knowing how to learn from television and radio. In the early stages of students experience with self- instruction, there should be a progression from a structured situation to a situation where students are able to organize material in their own learning package including more responsibility for deciding which areas to study, how to organize the study; and how to present it.
The programs should move from highly didactic to open ended; the structured learning should not limit the students learning so that students should do creative thinking. The presentation should avoid using many facts so that students find contexts and causal connections to create the students' ability to critically analyze what they see and hear and help them find their own way to knowledge. Emotional experiences should be provided. Student work should be based upon andragogical (adult education) principles.
Media can be used for learner interaction and feedback by providing for student drill and using techniques to motivate students to work and study; by actively involving learners through writing, talking, manipulating, competing, cooperating, critical viewing or activities on tape or in print components, or in some way respond to the teaching material to considerably increase learning effectiveness. Feedback should be immediate and timely to induce lesson submission; the assignment turn around time should be no longer than five days to increase student completion rates. Feedback should provide the correct response and a commentary on the incorrect response. The presentation sequence and rate should be learner controlled with branching to alternative units after incorrect answers. The instructional strategy should vary as a result of both current and past learner behavior and portions should repeat at the learner's volition. Students should have activities such as answering questions.
All student learning styles should be addressed as individuals may be primarily visual, auditory, tactile, conceptual, or quantitative in various combinations to focus on human learning and ensure learning for all students. Audio components should be provided for auditory students; visual components for visual students; and realia, models and other objects provided for tactile students.
Strategies should match student cognitive styles, previous experience and presentation factors. Cross-modal reinforcement should occur frequently where the same message is given through two modalities - words and pictures. Strategies should meet adult viewing styles which are open learners (about 33 percent who are interested in the world and learning, slightly older, more highly educated, who see television as one source of information), uninterested learners (50 percent of viewers who are not interested in learning, watch television for entertainment and have a low level of formal education) and instrumental learners (15 percent of population who are interested in learning as a means to a better job, young, upwardly mobile, blue collar or office workers, mid range in formal education, but do not consider television as a knowledge source).
Assignments should be specific to course content and may be created by students through the use of self-directed learning contracts. Assignments should help students become self-directed and adapt to local needs by utilizing faculty expertise through syllabus development and suggesting successful assignments for distance learners. Students should not be overloaded with more material than can be handled. Facilities should be available for laboratories. The first assignment should be due early, within 14 days, or within 40 days. There should be a great number of small assignments due rather than one project or several large projects, or one major assignment due each month. Computer marked assignments should be used.
ContentContent should be examined for appropriate scope of content; accuracy; authenticity; typicality; in good taste; reflective of research in learning; utilizes innovations in instruction; authoritativeness of materials; clarity; and illustrative of the interplay of process and growth of content. The same thing should be said more than once in different ways to replicate the central points. The course should be interesting and stimulating and provocative; lessons should be exciting to positively influence completion; and the video should have a long shelf life.
Differing viewpoints should be provided; controversial issues should be handled fairly without evidence of bias. The pluralistic society of multiple ethnic, racial, religious, social, geographic and sexual characteristics should be represented. The material should be relevant to today and the copyright should be recent and not older than two years. The material should be important and interesting to the learners.
Textbook
The textbook should be recommended by the producer; be acceptable; be as attractive as other textbooks to hold attention; be high quality, well presented and lavishly illustrated; be up to date; available on time; have further editions planned; have a clear role in course design; be widely used and the author's credentials should be appropriate and recognized. The textbook should encourage students to learn. The textbook should correlate well with other components and should match video revisions. If the text must be augmented a second text will have to be found or written if one is not recommended by the producer. If a reading anthology is recommended, it can be used to tailor the course to a particular focus by eliminating reading assignments.
Faculty Guide
The electronically mediated course should have a faculty guide to act as a mentor for the new instructor; provide in-depth discussion about instructional design; discuss content embodied in the components and how they relate to one another; present detailed teaching strategies and evaluation strategies; contain background information on course development, developers, consultants and advisors along with their credentials; and course goals. The guide should contain a course outline by lesson; weekly student activities for each week of the academic terms; test bank or suggested tests; alternative course structure; recommend varied uses of course materials; list required or suggested materials and sources; bibliography and sample promotion material. It should contain segments to guide students in learning from electronically mediated instruction, viewing holistically, finding patterns, developing analytical skills and other explanation about the course. If the faculty guide does not exist, local staff should have the experience to supply the necessary faculty support.
Human resources to support the course should be considered including whether the local instructor is competent and whether the course matches the instructor's teaching style. The instructor should write the course syllabus, assign additional readings, make assignments and grade them, hold an opening structured seminar, hold face to face meetings with individual students, call class meetings, maintain contact with students by mail, phone and meetings to add content for students' consideration; maintain student interest through study groups to provide support and raise completion rates. The instructor should be interested in and encouraging to the students. Technical facilities should be considered including, library access, physical circumstances and other logistical considerations.
A test bank should provide questions which are suitable for correspondence or proctored testing and based on the content. Viewing video programs should be linked to student assessment. Test keys should include a listing of where answers are found in the content. The test bank should have many types of short answer questions which can be graded by computer and suggest short essay questions. Test validity should be described. Students should be allowed to choose and provide evidence of learning.
Student Study Guide
The study guide should be recommended by the course producer and be acceptable to the institution. The guide should be an important component of the course which ties all course elements together to help the student complete the course. It should be written by content specialists as the course was developed and contain lesson-by lesson guides to meet course objectives, list additional readings, optional activities and be easily augmented by faculty by adding sections or deleting section depending on curriculum. Research shows that student completion rates increases by 10 percent if the study guide is written by the instructor. The guide should teach students how to use the course by explaining the function of the video and other electronically mediated content and give students guidance in what to look for and how to approach the program. It should train the student to look at video events holistically, to use analytic processes, what to focus on and how to discern patterns and self directed learning strategies. The guide should contain segments on objectives, components, lesson outlines, video outlines, glossary, key concepts, references, exercises and self-tests with explanations.
Pre-broadcast notes should be brief, but should clearly state the purpose of the program and what students are supposed to do before during and after seeing or hearing the broadcast or tape. Audio cassettes are not lectures but are tightly integrated with print to talk students through diagrams, illustrations, statistics or provide discussion material for analysis.
Computer Software
Recommended software should be suitable; easily available with appropriate site and home licensing at a suitable cost. Software is appropriate to content and used to present and test rule based procedures, areas of abstract knowledge where there are clearly correct answers so that educational objectives are achieved. Computers can be loaned to students. Logistics, including computer access to provide software to students should be suggested and the software should be available in many versions for many types of computers.
Computer Conferencing Software
For courses which include the ability for students and faculty to use computer conferencing by dialing into a central computer via modem, the choice of the computer conferencing software is important. E-mail systems may suffice, but it is usually better to have software specifically designed for teaching over the computer. Programs should be easy to operate by inexperienced students and faculty. It should not take as long to master the telecommunications program as it does to master the course content. Students may find that having to train on how to use a telecommunications program is an obstacle that they don't want to tackle. This is particularly true for short courses or continuing education courses. New programs are entering the market regularly.
Points to consider include: User friendly - pull down menus. Telecom-munications program can be downloaded by the student or mailed to the student. The computer program should support 2,000 or more students and faculty and the many messages they will generate. Ability to segment classes and allow entry only into authorized mailboxes. Ability to segment assignment mailboxes so that the flow of assignments and interaction can be easily followed. Ability to support private mailboxes for students and faculty, bulletin boards, faculty forums, students forums, registration, filing grades, delivering contracts to faculty and access by other administrators and interested lurkers. High points should be given to programs which dial the computer, upload new messages to appropriate mailboxes, download new messages and file them in existing files on the student/faculty members hard disk. Filing new messages is time consuming and this should have a high priority. The program should encourage the students to become interactive because a up to 40 percent of the students grade may be based upon this.
VideoVideo programs should use the full presentational power of video; words, still and moving pictures, events occurring in real time, slow or accelerated motion, animation and text. Production should be high quality as this correlates with lower attrition and higher grades particularly for borderline students. The technical quality should be acceptable or excellent, balanced and satisfying, meet professional standards or meet national broadcasting production standards; this is essential because of its motivational impact on students as the pleasure of watching the programs breaks the students' inertia of beginning to study. The video format should not differ too much from what is considered to be a good general commercial television program with an expensive appearance to compete with commercial television. Programs should be one-hour or can be shown as one-hour to meet normal programming times.
The number of programs should be high as more programs correlates with lower attrition. Tapes should be available for student loan as this has considerable advantage over a pre-scheduled distribution by cable. The video should not rely heavily on the lecture/talking head format or show students in a video class unless it is a teaching method class; the instructor should talk to the viewers for interaction.
Chemical experiments should be performed in an industrial laboratory to show the experiment's industrial application to demonstrate experiments or experimental situations where equipment or phenomena to be observed are large, expensive, inaccessible or difficult to observe without special equipment. The video should use the medium's unique possibilities to give students content that they would otherwise not get or see. The plot should not be wild or slapstick. The use of video material should be influenced by relevance more so than dramatic quality. Video is not used for dense, abstract ideas, comprehension of detailed arguments and facts; it is used to deal with abstract ideas through the use of concrete examples, stimulates sophisticated level of thinking which leaves interpretation and analysis open to the student. Programs should have structure, organization, sequential progression, be well paced to provide variety and a content development rate which holds attention and facilitates learning so that they are more swift than real life but not frenetic. Video should be used to increase the students' sense of belonging.
The video should demonstrate human interaction and time-space relationships to illustrate principles involving two, three or n-dimensional space; to act as a bridge between the concrete operational and formal, more abstract stages of learning; words (audio and written), dramatizations and music should generate attitudes and interest; use case illustrations, dramatization and supplantation (formulas, scope, rotation, animation, etc.) to advance content; complete coordination and integration between audio and video should exist; video should present unique material not found in the classroom; video should present well known content in unique forms; video takes society to the student to form links between class and life; video should use many open ended methods to encourage student inquiry; to change student attitudes towards a particular subject area by presenting material in a novel manner or from an unfamiliar viewpoint; and allow students to look into something otherwise inaccessible.
For student comprehension and instruction on how to approach television, video sequences should show the whole sequence, then repeat it with each sentence presented as a separate entity which is explained and elaborated upon; in later programs the elaboration should be decreased to give the student more independence. The video should encourage students to interpret, analyze and problem solve by facilitating the students' ability to apply knowledge, evaluate evidence or arguments, analyze new situations, bring insights to portrayed situations and suggest solutions. The camera work should be considered for appropriate and imaginative use of video which advances the content. Video should visualize the abstract to provide contrived images that present in visual form the concepts and relationships for which students cannot conjure images on their own. The screen should be used to its full potential with camera angles (single and two shots, point of view, over the shoulder, close-ups, wide shots and camera focus changes) and techniques (zooms, pans, swish pans, cuts) to attract attention through pictures, sound bites, demonstrations, diagrams and graphics. The video should show the world to create authenticity and effectively use color and motion. Effects should provide pace change and the material should dictate the use of effects such as wipes, freeze frames, flips, computer graphics, split screens; effects should not be used because the technology is available. Styles of clothing etc. should not detract. Clarity should be maintained by smooth bridges between segments and programs. Clear demarcations between discontinuous segments should be apparent in settings, presenters, etc. The use of sound should be considered so that sound, music and sound effects emphasize content. Sound should be imaginative, advance content, add variety and pace and not use a continuous music bed. Pictures are provided with clear verbal narratives for clarity.
The video instructor is important to the telecourse; is on camera; is competent; conveys interest in the content; transmits enthusiasm; and personality and appearance add to the effectiveness. The instructor does not lecture or preach; so that concepts are difficult to grasp and understand but simplifies the message by using understandable language, humor to motivate, make content palatable and act as change of pace; humor is situational, not slapstick. A diversity of experts, talent and characters provide variety and good acting with believable dialogue.
Costs
Costs should be considered as they relate to funding. Costs should be considered as to their appropriateness for a given media system and the proportion of money and resources to be devoted to various aspects of a media system; capital costs and recurrent expenditures, equipment obsolescence, staff, space and overhead, cost and delivery should also be considered. The cost effectiveness of the program to other programs on the same subject should be compared by projecting student per head costs and relationship to shelf life and student per head program costs to purchase and deliver. Media costs versus face to face instruction should be considered, broadcast costs versus loaned tape costs and other economies of scale where more students will make the media more cost effective.
from "A Technical Guide to Teleconferencing and Distance Learning," 3rd edition
