Evaluation of the Star Schools Projects
[This report has been divided into a number of separate web pages for browser-loading ease. You may view (and select) the contents by section title from the Contents below, or click on the "Next" button at the bottom of each page.]
Contents
The following is excerpted from the Star Schools Projects Report which was carried out by the Southwest Regional Laboratory (SWRL) and Abt Associates, Inc. (AAI). SWRL, 4665 Lampson Avenue, Los Alamitos, CA 90720 310-598-7661.
Carla Lane, Ed.D. Project Directorfor SWRL
Naida C. Tushnet
Christina Bodinger-de Uriarte
Diane Manuel
David van Broekhuizenfor Abt Associates Inc.
Mary Ann Millsap
Anne Chase
Introduction
The Office of Educational Research and Improvement (OERI) of the U.S. Department of Education has provided funding for three successive cycles of two-year Star Schools projects. At this stage in the development of the new communication-delivery technology, it is appropriate to assess the early efforts supported by federal funds.
The evaluation focuses on a series of questions posed in the authorization of Star Schools. The following questions address issues related to project organization and impact on students, staff, and schools.
Project-oriented:
- How effective are telecommunication partnerships programs and services after federal funding ceases?
- To what extend did projects use nonfederal funds? What were the sources of such funds?
- What activities were supported by grantees?
- Student programming
- Staff development
- Administration
Student-focused:
- How many students participate in Star Schools-sponsored activities? What are their demographic characteristics? Are there differences in services offered to economically and educationally disadvantaged and minority children? Are there differences in effectiveness of programming and services?
- How effective were Star Schools courses? Are differences in effectiveness related to differences in course objectives?
Staff-focused:
- How were studio teachers selected? What support did they receive?
- How were the staff members responsible for distance learning at the school site selected? What support did they receive.
- What staff development programs were offered? To how many teachers? With what effects? How much time was spent in staff development?
School focused:
- What were the socioeconomic and geographic characteristics of participating schools? What types of programming were offered to different schools?
- What were the effects of distance learning on curriculum and staff patterns at participating schools?
Evaluation Approach
Conceptual Framework
The evaluation focused on two aspects of the context for the Star Schools Program. First the federal context influences the organization of the Star Schools projects as well as the services they provide. The federal context includes not only the legislation but changes in priorities For example, when the authorizing legislation was passed in 1988, the National Education Goals had not yet been formulated. They are now significant objectives for Star Schools activities. One consequence is that science and mathematics programs developed during the first cycle of funding have a somewhat different content and focus from those developed later.
Legislative requirements form the second aspect of the federal context. Some of these have not changed from the inception of the program. For example, the law mandates that projects be multistate consortia and have at least three partners. These requirements had an impact on the way projects were organized and the issues and problems they faced. In addition, the requirements affected the functioning of distance learning activities in the funded organizations after federal funding ceased. The multistate consortium requirement meant that projects confronted issues of teacher certification, as well as coursework and testing requirements, across state jurisdictions. Funded projects addressed these issues in a variety of ways that involved operations, series delivery, and the targeting of activities. In addition, the statute requires at least one half of the schools served to be Chapter 1 schools. (Chapter 1 is the single largest federal elementary and secondary education program. It is designed to provide supplemental services to low-achieving students in low income schools.)
The second set of contextual issues comprises state and local characteristics. These issues had an impact on the effectiveness of Star Schools projects in several ways. First, some state regulations, particularly those that raised high school graduation on state university entrance requirements, helped create a market for distance learning projects. Isolated rural schools employ distance learning technologies to ensure opportunities for their students and to comply with state regulations. Second state and local regulation created barriers to the effectiveness of distance learning technologies. For example, different requirements for certification in a subject area in different states and regulations or contract agreements that require the presence of a teacher certified in the subject being taught in the classroom on occasion precluded participation in Star Schools activities. At the minimum, certification issues required projects to develop agreements with a number of states.
Local contextual issues include demographic and geographic information. Perhaps more important, however, the local context provides academic and other experiences for students and teachers that influence the effectiveness of distance learning programs. For example, the local context provides students with prior opportunities to learn. Students who come to advanced physics classes, for example, well- prepared in trigonometry and calculus, are more likely to succeed than students who take physics with weaker preparation. Local context also includes the qualification of teachers who prepare students and those who support distance learning activities.
The effectiveness of Star Schools activities that are integrated into existing coursework or are designed to improve teaching are particularly influenced by local context. Student learning in courses that combine school-site and distance learning opportunities,, for example, is likely to vary depending on the quality of teacher implementation of curriculum and instruction. Similarly, the effectiveness of activities aimed at instructional improvement will be influenced by improvement efforts at the local site as well as other staff development opportunities available to teachers.
The contextual issues also influence the organization and service offerings of the Star Schools projects. Of particular interest within the conceptual framework are the project-focused, staff-focused, student-focused, and school-focused questions raised by Congress and OERI. The framework provides a method for organizing descriptive information about such matters as the number of students served, the types of services they receive and demographic and geographic distribution of services in a manner that facilitates analyzing relationships between those matters and the context and condition under which services are received. It also enhances analysis of the relationship of the objectives of particular activities to the distribution of services and to their effectiveness in achieving their own goals.
Various configurations of Star School services affect students' access to learning opportunities. Indicators of access are curse offerings (the number and content), student enrollment, and student access to quality teaching.
Finally, the outcomes are student learning and improved instruction.
Methodology
The evaluation has two phases, each with a distinctive methodology. Phase 1, from which this report results, relies primarily on qualitative data collection procedures. Phase 2, which began in fall 1993, combines survey research methods with case study research.
Phase 1 was designed to yield defensible understandings of the existing Star Schools projects. It drew on the following data sources:
- a literature review encompassing research findings from distance learning related studies and of alternative approaches to achieving the objectives addressed through Star Schools activities.
- site visits to Star Schools projects and schools and to other distance learning projects; and
- project reports.
Project agendas and site protocols, adapted from procedures described by Miles and Huberman (1984), provide a means of integrating the data from the various sources. Project reports were analyzed using a content analysis procedures that provided quick retrieval of project-generated information about each element in the conceptual framework (Marshall & Rossman, 1989). The information was then included in project agendas, which, in turn were used to generate site protocols. The site protocols identified the individuals to interview and events to observe during the site visits to Star Schools projects and schools. One use was to highlight "missing" information that must be gathered on site. Another was to indicate the questions that probed more deeply into project activities organization, and impact than was possible from written documents. During the analysis phase, the project agendas provided a means of integrating qualitative data collected on site and quantitative data gleaned from project documents.
The first phase of the study involved data triangulation; that is, collecting information from multiple sources using multiple methods. Staff collected descriptions of activities from project-generated documents; interviews with project staff, staff from participating institutions, and the recipients of services; and observations of the activities.
Two person teams conducted the site visits. They spent at least four days on site, two at project headquarters and one at each school. The sites visits had three purposes:
- They provided the opportunity to confirm
and extend the information gleaned from project documents.
- They yielded information that addresses
the evaluation questions associated with project organization
perceived effects of federal requirements, and actual school-level
experiences with Star Schools The site visit information was
particularly important in addressing questions about the effectiveness
of telecommunications partnerships programs and services after
federal funding ceased.
- The site visits served to focus Phase 2 data collection and analysis plans.
In addition to interviews with project staff and teachers and administrators at school sites, staff observed Star Schools classes. The major purpose of the observations was to gain insights into the degree to which students have the opportunity to interact with one another and with the distance learning teacher, student responses, and the role of the on-site facilitators.
The information collected on site was synthesized by the site-visit team following its visit. Team members reread interview notes, notes from observations and project documents, coding information according to the conceptual framework. They prepared an interpretive summary of their findings, which served two purposes. First, it provided a concise statement of the progress and problems of particular Star Schools project. Second it contained tentative analyses that explained the status of the project. In analytic meetings site visitors reviewed the interpretive summaries to develop what Yin (1981 calls "causal arguments" both within and across cases. The causal arguments were used to identify the existence of phenomena in more than one case under predictable conditions. For example, a preliminary analysis revealed that the use of taped, as contrasted with live, broadcasts was related to the structured broadcast schedules, which met project needs to provide regular service, but conflicted with in-school schedules. The causal argument is that the combination of rigid technologies and organizational inertia leads to creating flexible use of programming. The analytic meetings followed procedures recommended by Miles and Huberman (1984), which culminate in conclusions about Star Schools and its activities. These conclusions were then framed in terms of the literature in completing this preliminary interim report.
Limitations of the Report
This report accurately reflects current information about the Star Schools Program It provides descriptive information about project operation the numbers and types of students and schools reached, and the role of the on-site facilitator and teacher. Equally important, the report contains information about how Star Schools projects spent federal funds and how funds from other sources were used to extend the implementation of the activities. It also provides policy-relevant information about how Star Schools distance learning is used at school sites, the ways such programs are combined with others at some sits, the perceived value to students and schools, how programs are combined with others at some sites, the perceived value to students and schools, how federal and state context influence the use of distance learning at the school level, and reported effects on the organization of curriculum and instruction. In addition, the report includes recommendations for policy and future study well-grounded in the experience of both providers and recipients of Star Schools services. The report also includes an evaluation of the relationship of federal requirements to the operation of the program and recommendations for potential changes in how the program is organized. In sum, this first year of a two-tear study provides useful knowledge to policy makers as they consider the value of the Star Schools Assistance Program and ways to enhance its positive effects.
At the same time, the report has limitations, all stemming from its timing. First although site visits were made to all Cycle One and Two projects, at least two Cycle One projects kept few records of the Star School years, and one was no longer providing distance learning opportunities to schools. Consequently, information about those projects is more limited than information about others. Also, the remaining projects had adjusted programming to accommodate the end of federal funding. While such adjustment constitute important information included in this report, they also limit the evaluators' ability to see Star Schools as it originally operated. Second, the report relies heavily on project-generated evaluations, which varied greatly in quality and perceptions of Star Schools project staff and teachers who received services. Consequently, although the report provides information about probable impacts on teachers, schools and students, it will be important to gather independent data about those impacts. It is equally important to gather data relating impacts to federal and state contexts, project organization and processes, and the type of distance learning activities. These will be pursued during Phase 2.
Finally the report includes some speculation about how the impact of Star Schools distance learning can be enhanced. Phase 2 will explore these speculations more systematically.
Despite its limitations, the report has many values. It contains clear descriptive information, documented associations, and grounded policy recommendations. Although an interim report, it goes farther than previous research to document and evaluate Star Schools distance learning, its possibilities and problems.
