Evaluation of the Star Schools Projects
[This report has been divided into a number of separate web pages for browser-loading ease. You may view (and select) the contents by section title from the Contents, or click on the "Next" button at the bottom of each page.]
Findings
State Regulations and Requirements
State regulations and requirements had both positive and negative effects on projects. Some state requirements, notably those related to foreign language, helped create a demand for distance learning courses. However, some requirements may limit schools' willingness and ability to participate in Star Schools activities. On balance, state requirements had more positive than negative effects on Star Schools.
State requirements positively influenced participation in Star Schools foreign language courses but may have had a negative impact on participation in other courses. Two projects cited either high school graduation or state university admissions requirements as creating a market for Star Schools services. However, one developer of supplementary materials reported that teachers in one state were unwilling to use them because they feared not covering material found on the college admissions test.
Generally speaking, teacher certification and time allocation requirements were only an issue for whole courses in which school-site personnel are not certified in the same area. These did not serve as major barriers to Star Schools activities. Most projects used certified teachers as whole-course studio teaches and worked with other participating SEAs to gain reciprocal certification, including, in one case, submitting studio teachers to a blood test required by another SEA. Projects that used university teachers as studio teachers required that classrooms be supervised by certified personnel, preferably in a subject close to that being offered. This solution enabled schools to participate, but served to limit the number of sites interested in the programs, according to project personnel.
One state required that classrooms, as well as studio, teachers be certified in the subject being taught. This served as a major limitation on distance learning opportunities in that state.
Course approval did not present a problem except in two cases. The same state that required a certified teacher within the classroom rigidly applied requirements that classes meet for a defined time period, and only counted the on-air time, as did another state that refused to participate in activities designed to supplement classroom instruction offered by another project. To deal with issues of course approval, SERC noted that courses developed by teachers were sometimes reviewed by SEA specialists for approval, particularly if they were to be used to fulfill a graduation requirement.
Selection of Studio Teachers
Studio teachers were selected based on referrals, interview, and screen tests whether they were responsible for entire courses or for supplemental instruction. Those teachers who had gone through an interview and screen test prior to selection were usually more successful than those who were hired only on the basis of reputation or interest and whether they were selected from among classroom teachers or university professors. When screen tests were not being used, either some teachers were replaced or course enrollments dropped.
Support for Studio Teachers
All projects using satellite transmission reported that studio teachers received support from studio production teams, particularly in preparing visual aids and graphics. As one studio teacher reported, "I had to learn from the people with a video orientation how to think in images and use pictorial things to enhance my teaching style." Further, all but two projects also reported that studio teachers received some support from curriculum specialists or other resource people. Those projects whose studio teachers did not receive support on the content of instruction were those that relied on university professors.
Selection of School-site Personnel
The criteria for selecting school-site personnel varied with whether the distance learning was designed as whole-course or supplemental instruction. For whole-course instruction, the duties of school- site personnel focused on the logistics of distributing materials and classroom management. Unless states squired school-site personnel to be certified teachers, the on-site facilitators typically were support staff and non-certified teachers. However, the Midlands project encouraged schools to use certified math and/or science teachers for their physic and calculus distance learning courses, and project staff believed the courses were more effective with such school staff.
With supplemental instruction, the regular classroom teacher had to be in the classroom for the distance learning activities. In these projects, Star Schools personnel provided some technical assistance to these teachers, who, for the most part, had volunteered to participate. In all projects, selection of school site staff was at the discretion of individual schools or school districts; Star Schools project staff played no role in the selection of on-site staff.
Amount of Assistance Available to Individual Schools
The amount of assistance available to school-site personnel varied from project to project, with the tasks that school-site personnel undertook, and with whether the Star Schools project provided equipment. Among those projects providing whole-course instruction, assistance ranged from short (half-hour to hour) telecourses on the specific technology and general facilitator training to intensive technical support. Such intensive support included project technicians who traveled to receiving sites when technical question could not be answered by telephone. Some projects have a school hotline, outreach visits, and quarterly calls to facilitators and schools. STEP/Star and SERC placed greater emphasis on local assistance and were well regarded by sites visited than did TI IN, Midlands, and BCSN. Interactions between the school-site staff and studio teachers usually depended upon the initiative and interest of the school-site staff. Formal structured interactions were infrequent, but there are cases of extensive cooperative and communication.
Generally speaking, the amount of staff development was greater when projects focused on supplemental instruction rather than on whole-course instruction. MCET, for example provided far more staff development than any other Star Schools project. All teachers participated in several one-day conferences, in a one-week residential summer institute, and in two teleconferences. Moreover, two-full time coordinators each worked with 30 schools and were available for substantial on-site technical assistance. TERC, the only project relying on computer networking, used a training pyramid, where TERC staff trained university-based staff who, in turn, worked with individual schools. Problems with the computer technology, however, precluded attention to integrating the content of the technology into classroom instruction. One supplemental project served so many schools that it was too large to provide much personal assistance to individual teachers, relying instead on print materials and teleconferences. Although each participating district had a full-time district-level person responsible for recruiting and assisting schools, almost 100 schools per district, on average, participated in the program, thus rendering even local assistance minimal.
School-Focused
Schools participate in distance learning projects for a variety of reasons, according to interviews with school personnel in the schools that projects recommended as examples of "best-practice" implementation of Star Schools activities. Why schools participate may, for example depend upon:
- state graduation or college-entry requirements that otherwise would be difficult to meet;
- desires to increase curriculum offerings available to students;
- efforts by school staff to enhance educational experiences for students;
- beliefs in the importance of exposing students to people living in other settings;
- desire to expose students to multicultural educational experiences;
- efforts by school staff to broaden parent and community participation in schools; and
- intentions to increase both faculty and student access to and comfort with technology.
Use of Technology
For the Star School projects funded in the first two cycles, the dominant distance learning technology was via satellite. Districts or schools used satellite dishes as downlinks to relay live studio broadcasts into the classroom(s) Project staff members do not know the extent to which satellite broadcasts were taped and not viewed live, although estimates range from 50 percent to 90 percent.
To enhance interactivity, each project provided an "audio bridge" --a single or series of telephone lines to connect classrooms directly with the studio teacher during the broadcast. Most (but not all) participating classrooms had a telephone. Some projects used the audio bridge on a first-come-first-served basis, while others rotated systematically among schools so that each school or classroom would be guaranteed "on-air" time. The extent of interaction often depended on the initiative of the on-site facilitator. The amount of interaction also was a function of the total number of schools enrolled in the course or module; the larger the enrollment, the less access to on-air time.
Some projects also used the audio bridge immediately after a program aired, during specific "office hours" for the studio teacher and/or during prearranged tutorials with other staff. Audio-bridge time also was enhanced during special event broadcasts so that incoming calls were directed to several professionals in addition to the studio teacher.
The extent to which other technologies were used is in part related to program objectives. Whole- course instruction always used satellite technology that was usually enhanced with an audio bridge, computer networking (particularly electronic mail), compressed data transmission equipment, or fax machines. Supplemental instruction designed to enhance classroom teaching typically used more varied technologies.
MCET provided a menu of multiple technologies from which participating middle school teachers chose that was most applicable to their school, and all schools were provided with the needed equipment to use the technologies. With a variety of technologies available and ample on-site technical assistance provided, the patterns of use by MCET teachers may inform future educational applications of technology.
The MCET report on teachers'' use of technology indicates heavy reliance on those tehnologies that teachers themselves could schedule into their classroom day - computers telephone and television . Satellite programming, althought a major focus of the MCET Star Schools project, was used far less intensively than any computer-related technology. It outranked only the fax machines and the laser disc player in frequency of use. The uses of the laser disc player may be an underestimate because some teacher indicated that with so much to learn during the first year of operation, tjey had postponed learning how to use the laser disc player until after the grant period ended.
TERC offered a variety of computer-based technologies and was the only Star Schools project during either Cycle One or Two that did not use satellite programming. TEAMS' technology more closely resembles that of whole-course instruction; it provides satellite programming with an audio bridge and extensive classroom materials for its supplemental math and science modules.
Satellite-based and computer-based technology applications differ in two fundamental ways. First, the satellite-based distance learning activities are generated from a central point and sent out to receiving sites while computer-based applications frequently included information sharing that begins at the school site Second, satellite-based distance learning activities are scheduled by the provider. When school personnel wish to use such programming live, they must schedule other courses and activities around the broadcast. In contrast, the use of computer-based technology can be scheduled within a classroom. These differences seem to influence the choice of technology at the project level and how Star Schools distance learning is implemented at school sites.
Student Outcomes
Three student-level effects of distance learning are of interest. First is the effect of distance instruction on learning. This can be demonstrated through outcome data such as passing grades in the courses, gains in achievement, and good performance on tests. Second is the effect of distance learning on student interest and motivation, for which only indirect outcome indicators are available. Third is the effect on student work as evidenced by special products.
Four projects did not collect outcome data of any kind. Project staff referred to the need for longer treatment periods before one could expect to perceive impact on students. Two projects gathered information on student outcomes using a single method of assessment and the other two used two methods.
- One project reported grade distribution
across their distance learning courses.
- Two projects formally assessed student
performance through comparison tests with studentsreceiving traditional
classroom instruction. A third project used pre-and post-tests.
In addition, a single site, a fourth project looked at change
in score on norm-referenced tests.
- Two projects surveyed teachers concerning
the impact of distance learning on their students.
- Three projects surveyed or interviewed students to obtain feedback on the distance learning experience.
The project reporting grade distributions offers whole course high school instruction in AP level subjects, foreign languages, and specialized math topics. In this project, 88 percent of the distance learners earned a C or better in the broadcast course. Almost half (42 percent) of the students earned As. No comparison data are provided.
In both projects that used comparison tests, project staff members remain dissatisfied with the test. In one case they believed although offering courses in the same content area, traditional classroom instruction and the distance learning instruction were not actually comparable on a specific level of subjects covered. Students in distance- earning classes demonstrated lower levels of performance in all areas tested. In the second case, distance learning groups performed better than their traditional classroom counterparts. However, items were changed between the pre- and post-test period, and 40 percent of the students did not finish the tests in the time intended, but returned to complete them . A third project used pre- and post-tests, demonstrating a 45 percent gain. Comparison groups were not used. One site looked at norm-referenced test scores of the participating Chapter 1 students and found gains for 16 of the 24 middle school students.
Teacher feedback was collected through formal interviews and/or surveys by two projects. In both instances, teacher reports were favorable. Teachers in one project reported that academically disadvantaged students were better served distance learning than by traditional classes. They also reported that minorities and learning disabled students had higher rtes of attendance during distance learning module days. Teachers in the other project also reported high rtes of attendance, but did not differentiate among types of students most affected. Teachers also reported that the hands-on approach resulted in observably higher levels of student motivation during broadcast instruction.
Student feedback also was elicited by two projects. Students in one project reported no significant differences between distance learning and traditional classroom learning: class cohesiveness, goal direction, teacher characteristics and skills, and teacher support. The other project had students complete course-specific evaluations. Across courses, 43 percent of the students said they believed they learned as much in Star Schools classes as they would have learned in traditional classes.
In addition, anecdotal illustrations of successful distance-learner performance were offered by a number of teaches, principals, and the project staff across all eight projects. For example, in two projects, AP studio teachers reported that they believed AP distance learners were passing AP tests at the same rate as those taking traditional classroom AP courses. The staff of one project pointed to the fact that three distance learning foreign language students were among the top 10 finalists in a statewide Japanese competition. Another foreign language project reported project-taught students "sweeping" the site language competition for five years. Another project reported spontaneous student use of languages learned via broadcast.
Anecdotal information also is used to indicate strength of motivation among distance learners. One site reports that students drove up to 35 miles each way to receive broadcast instruction in algebra during the summer. Another site reports that students voluntarily gave up 15 minutes of their daily lunch period to receive broadcast instruction. Classroom teachers at several sites also stated that student attendance rates were better on broadcast days than otherwise. They attributed the increase in attendance to the teaching methods used during the broadcast module.
Outcome data are sketchy. However, what data are available suggest that for at least some courses, learning takes place, students are motivated, and work is accomplished. This also is the general opinion offered by project staff, principals, and classroom teachers and facilitators.
Staff
Those programs offering supplemental instruction, most notably TEAMS and MCET, were intended to change teacher' attitudes and behaviors in the classroom. Both TEAMS and MCET conducted surveys of participating classroom teachers on changes in instructional practices. In TEAMS, a strong majority (86 percent) of teacher responding to the internal evaluation survey reported using different instructional materials as a result of participating in the distance learning instruction.
The internal evaluation surveys of MCET middle school teachers do not reveal appreciable changes in their views on science teaching and learning. They already supported hands-on science, interdisciplinary teaching, and cooperative learning prior to participation in project activities. Teachers appeared philosophically to have already taken a cognitive or constructivist approach to science education. However, teachers reported changes in their classroom behavior through participation in the Star School activities. They indicated an increase in the use of multiple technologies in the classroom (95 percent of teachers), innovative ways of teaching science (85 percent), interdisciplinary teaching (71 percent), cooperative learning (68 percent), hands-on science (57 percent) and team teaching (47 percent). Teachers also reported changes in classroom management strategies, with more emphasis given to organizing classes with small group activities (85 percent of teachers) and less given to lecturing to the whole class (69 percent). Classroom observers noted that the multiple technologies and hands-on activities created interest among students and that students were more responsible for their own learning. Observers also noted that teachers still had a long way to go in asking students for explanations, having students use previous experiences to explain concepts, and planning how to group student to ensure maximum learning (Drexler & Kapitan, 1993).
Although there is self-report of changes in attitude and behavior, in neither project were changes reported in the staffing patterns in the schools.
Those distance learning projects providing whole-course instruction did not explicitly intend to change individual teacher attitudes or behavior. The adults in the classroom often were non-teaching staff whose primary responsibilities in the distance learning classroom were for managing the classroom and distributing materials. In a few schools, there was anecdotal information that some facilitators who were already certified teachers improved their teaching skills, and there were reports that certified math teachers who supervised calculus courses became comfortable enough with the material to teach independent of the distance learning course.
Schools
Star Schools funded activities had two types of effects on curriculum and instruction within receiving school. First, it broadened curriculum offerings and, in specific instances, built local capacity to provide instruction in a greater array of courses. Second, Star Schools projects sponsored activities designed to improve instruction by providing models and an array of curriculum materials, and these had different levels of impact. Following is a discussion of each type of school-level impact.
Although the major purpose of activities that increased the number and types of courses offered was to provide students with increased educational opportunity, instances exist in which the courses increased local capacity for curriculum breadth. For example, the Midlands project required a certified mathematics teacher to serve as the classroom facilitator for the calculus course. The project reports approximately 10 cases in which the classroom teacher became comfortable enough with the subject to offer the course independent of distance learning.
The more significant impacts on school curriculum and instruction were intended from activities designed to influence how teachers and students interact during regular classroom instruction. The actual effects, however, varied according to the design of the activities and how well the met local needs. For example, a major "success story" came from a site that had been searching for ways to integrate curriculum and revitalize long-term teachers. The school had little money for materials (less than $2 per pupil per year for science materials), but had a group of teachers who were looking for ways to stimulate themselves and their students. The teachers wrote the required proposal to the Star Schools project for participation and took "everything we could get our hands on." During the project years, they implemented most of the offered activities, thereby changing science instruction to a "hands-on" approach. After funding ceased, they m modified some activities and continued others. Perhaps most strikingly, one teacher had postponed implementing some activities during the grant years, felling unprepared to tackle the full array of project activities, but planned to implement them subsequently.
School staff in other projects reported that non-participating teachers borrowed materials and approached curriculum changes in ways that were congruent with the Star Schools activities. Because the schools also were participating in other site and district programs designed to increase cooperative learning and problem-solving approaches to teaching, it is impossible to determine how much the existence of Star Schools activities contributed to teachers' interest in the materials. Nonetheless, the availability of materials was helpful to local efforts.
Activities directed at changing instruction in regular classrooms can provide models of instruction and supplement existing curricula. However, they face a dilemma. On the one hand, what is known about teacher development (Loucks-Horsley & Associates, 1990; Joyce & Showers, 1988) requires close interaction between the staff development provider and the classroom teacher. On the other hand, the strong argument made for using distance learning technologies for staff development focuses on its low cost per teacher. That is through distance learning, large numbers of teachers can be reached at a single time. However, if the studio teachers, who provide model of instruction, also are expected to coach the classroom teacher, they must have knowledge of how each teacher understands and uses the preferred modes of instruction . The needed intimacy, then, may require a limited number of participating teacher, unless the distance learning is accompanied by on-site support. But either limiting the number or providing on-site assistance means that the per-teacher cost may be no lower than other approaches to staff development. As a result, using distance learning as a vehicle for improving instruction may not be cost effective when compared to other ways to accomplish the same objective.
