TEAMS Evaluation 1993-1994
Conclusions & Recommendations Summary
[This article has been divided into a number of separate web pages for browser-loading ease. You may view (and select) the contents by section title from the Contents, or click on the "Next" button at the bottom of each page.]
Planning and Evaluation Project (PEP) Sites
Each school which used TEAMS (Apollo 2000) was asked to nominate itself as a Planning and Evaluation Project (PEP) Site. Regional TEAMS Coordinators worked with schools to complete the nomination forms. Nomination forms required data on demographics of the school and agreement to in-depth study techniques including the site visit, if they received the nomination. Each region nominated several more schools than were required so that the evaluation team could make the final PEP school selection based on demographics that reflected diversity. Originally 35 sites were to be selected, but two others were added (one in Washington, DC and one in Utah) bringing the total to 37 in-depth study sites.
Methods
Information and data were collected for the evaluation through two primary methods, survey instruments and PEP site visits. Survey instruments were developed for administrators at the PEP in-depth study sites, teachers at PEP in-depth study sites, TEAMS teachers at regular sites, TEAMS regional coordinators, and Pacific Mountain Network (PMN) public television station affiliates.
One of the requirements for a PEP site was that the survey instruments be returned electronically using TEAMSNet. TEAMS (Apollo 2000) provided each PEP site with a modem and software program. Because of technical problems, very few questionnaires were returned electronically. Several schools did not have Microsoft Word software or a Macintosh computer. Some sites could not easily access a telephone line for their modem.
Site visits were made to the 37 PEP School sites during January, February, March and April of 1994. At each site, focus interviews were conducted with principals, teachers, resource teachers, other involved staff and, at some sites, with the students. Classroom use of a TEAMS program was observed where possible. TEAMS programs were not always being broadcast or used on the day of the site visit. Most site visits were conducted in the classroom where the students used TEAMS as substitute teachers were not available.
PEP Sites
Pep sites were located in Los Angeles County, Los Angeles, Boston, Detroit, St. Louis, Kansas City, Washington, DC, Utah, Arizona and Missouri.
Respondents
RESPONDENTS
#
PEP School Administrators 37
PEP Site TEAMS Teachers 113
TEAMS Teachers 330
TEAMS Regional Coordinators 8
Pacific Mountain Network Affiliates 9
Students 7,299 Total Responding 7,796
Summary of Results
A TEAMS (Apollo 2000) Student Progress Form was sent to every TEAMS teacher as part of the survey instrument. The form collected basic information on gender and participation in Chapter 1, LEP, Gifted, and Special Education programs. The form asked the TEAMS teacher to describe the degree of the outcome for each student that could be attributed solely to using TEAMS. The scale of one to four was used where four was a great degree and one was none. Teachers ranked the 7,299 students on the following attributes.
- Improved Content Knowledge and Skills
- Improved Critical Thinking and Problem Solving
- Improved Language Skills
- Increased Interest in the Subject Area
- Improved Quality of Work
- Increased Interest in School
- Improved Attendance
- Improved Behavior
- Takes Responsibility for Own Learning
- Greater Confidence as Learner
- Higher Self-Regard
Usable data was returned for 7,299 students, making this the largest study conducted with students participating in a Star Schools Project. It is also the largest study conducted of students in a distance learning program during this period of use of technology for education. While other studies of student learning have generally focused on whether students learn as much or as well from educational technology as from traditional classrooms, the TEAMS evaluation focused on students using the same programs and the same technology, but studied other educational outcomes as an indication of the success of the TEAMS (Apollo 2000) program.
The evaluation also focused on new ways to gather information about student learning outcomes. While it has become apparent that students learn as well from educational uses of technology, the use of standardized test scores does not report all of the learning that is taking place. It is important that the developers of standardized tests begin to include ways to measure the new skills that students are learning, that society values, and are mandated through the Educate America Act: Goals 2000.
The TEAMS student progress form created for the TEAMS (Apollo 2000) Star Schools Project endeavored to develop a significant new method to collect and assess student learning and improvement through distance learning programs. The form was successfully used by teachers and will be used throughout the 94-96 evaluation of TEAMS (Connections 2000) Star Schools Project.
Results of Student Progress Attributed to TEAMS (Apollo 2000)
A series of multiple linear regressions were performed on the data. For all of the student results, no noted change in a variable did not mean that the student was working at an F grade level. TEAMS programming is seldom used as the only source of mathematics or science education in the classroom. Teachers may have been unable to see a change which was directly attributable to TEAMS programming and thus recorded a score of one (no change due to TEAMS) rather than a score of two to four showing change due to TEAMS. For example, many teachers noted on their survey instruments that students always came to class and there was little room for improvement in the question about increased attendance. Other teachers noted that the behavior in their classrooms was always good and thus, a score of one for no change, was not to be a considered a bad mark or a poor reflection on TEAMS. Simple regressions were conducted using "improved content knowledge and skills" as the dependent variable. All independent variables were significant at the level of P=.0001 (see Table 1).
Table 1.
Simple Regressions on the Dependent Variable Improved Content Knowledge and Skills Independent
VariableCoefficient
ErrorStd.
Coeff.Std.
Valuet-
ValueP-
ValueIntercept 2.944 .013 2.944 223.680 .0001 M/F -.038 .019 -.024 -2.025 .0429 Intercept 3.010 .013 3.010 233.948 .0001 Ch 1 -.179 .019 -.111 -9.564 .0001 Intercept 2.976 .011 2.976 271.799 .000 LEP -.189 .021 -.104 -8.915 .0001 Intercept 2.895 9.810E-3 2.895 295.069 .0001 Gifted .332 .032 .120 10.303 .0001 Intercept 2.950 9.700E-3 2.950 304.092 .0001 Special Ed -.314 .033 -.112 -9.639 .0001 Intercept .625 .019 .625 32.476 .0001 Thinking .807 6.481E-3 .824 124.440 .0001 Intercept 1.504 .024 1.504 63.633 .0001 Language .541 8.514E-3 .597 63.493 .0001 Intercept .942 .025 .942 38.164 .0001 Subject Interest .659 7.901E-3 .699 83.449 .0001 Intercept 1.432 .024 1.432 60.148 .0001 Quality .563 8.534E-3 .611 65.992 .0001 Intercept 1.629 .024 1.629 67.763 .0001 School .490 8.587E-3 .555 57.036 .0001 Intercept 2.352 .021 2.352 112.473 .0001 Attendance .269 8.889E-3 .334 30.246 .0001 Intercept 2.183 .022 2.183 100.669 .0001 Behavior .330 8.847E-3 .400 37.330 .0001 Intercept 1.645 .023 1.645 70.448 .0001 Responsibility .494 8.483E-3 .563 58.194 .0001 Intercept 1.502 .024 1.502 62.995 .0001 Confidence .531 8.424E-3 .594 62.992 .0001 Intercept 1.596 .024 1.596 67.678 .0001 Regard .503 8.441E-3 .572 59.612 .0001 In the multiple linear regression which used "improved content knowledge and skills" as the dependent variable, ten independent variables were found to be significant at P= .05 and above. Improvement was significant for Chapter 1, LEP and Special Ed TEAMS students. Teachers throughout the US. saw improvement for these students in improved critical thinking and problem solving, language skills, increased interest in the subject area, improved attendance and behavior, and improvement in the responsibility the students took for their own learning as well as a positive increase in student self regard (see Table 2).
Table 2.
Results of Multiple Linear Regression on Content Knowledge and Skills Variable
P
Chapter 1 P=.0005 LEP P=.0072 Special Ed P=.0113 Improved Critical Thinking
and Problem SolvingP=.0001 Improved Language Skills P=.0001 Increased Interest
in the Subject AreaP=.0001 Improved Attendance P=.0001 Improved Behavior P=.0324 Takes Responsibility
for Own LearningP=.0333 Higher Self-Regard P=.0019 A multiple linear regression was done using "increased interest in the subject area" as the dependent variable. Nine independent variables were significant in this procedure.
TEAMS teachers saw improvement for special education and gifted students in improved critical thinking and problem solving, quality of work, increased interest in school, improved attendance and behavior, and improvement in the confidence students had in themselves as learners (see Table 3).
Table 3.
[Increased Interest in the Subject Area Intercept P=.0000]
Results of Multiple Linear Regression on Increased Interest in the Subject Area Variable
P
Gifted P=.0001 Special Ed P=.0206 Improved Content
Knowledge and SkillsP=.0001 Improved Critical Thinking
and Problem SolvingP=.0001 Improved Quality of Work P=.0001 Increased Interest in School P=.0001 Improved Attendance P=.0001 Improved Behavior P=.0177 Greater Confidence as Learner P=.0001 To determine specifically what improvements teachers were attributing to the four groups of students, multiple linear regressions were performed for each of the four groups individually (as the dependent variable) and using the twelve assessment criteria as independent variables.
For Chapter 1 students, the independent variables that were significant were improved content knowledge and skills, improved critical thinking and problem solving, improved language skills, improved quality of work, improved attendance, and a higher self-regard (see Table 4).
For gifted students, the independent variables that were significant were gender, improved critical thinking and problem solving, improved language skills, increased interest in the subject area, improved quality of work, increased interest in school, improved attendance, improved behavior and a higher self-regard (see Table 5).
For limited English proficient (LEP) students, the independent variables that were significant were improved content knowledge and skills, improved critical thinking and problem solving, improved language skills, increased interest in the subject area, improved quality of work, improved attendance, an increase in taking responsibility for their own learning, and a higher self-regard (see Table 6).
For special education students, the independent variables that were significant were gender, improved content knowledge and skills, improved critical thinking and problem solving, an increase in taking responsibility for their own learning, and a greater confidence as a learner (see Table 7).
The intercept P on each of these groups of students shows that the TEAMS (Apollo 2000) program is significantly impacting the learning of these students.
Table 4.
[Chapter 1 Students Intercept P=.0001]
Results of Multiple Linear Regression on Chapter 1 Students Variables
P
Improved Content
Knowledge and SkillsP=.0001 Improved Critical Thinking
and Problem SolvingP=.0001 Improved Language Skills P=.0001 Improved Quality of Work P=.0004 Improved Attendance P=.0403 Higher Self-Regard P=.0001
Table 5.
[Gifted Students Intercept P=.0525]
Results of Multiple Linear Regression on Gifted Students Variables
P
Male/Female P=.0724 Improved Critical Thinking
and Problem SolvingP=.0001 Improved Language Skills P=.0255 Increased Interest
in the Subject AreaP=.0001 Improved Quality of Work P=.0001 Increased Interest in School P=.0010 Improved Attendance P=.0025 Improved Behavior P=.0463 Higher Self-Regard P=.0031
Table 6.
[Limited English Proficient (LEP) Students Intercept P=.0001]
Results of Multiple Linear Regression on LEP Students Variables
P
Improved Content
Knowledge and SkillsP=.0001 Improved Critical Thinking
and Problem SolvingP=.0004 Improved Language Skills P=.0001 Improved Quality of Work P=.0019 Improved Attendance P=.0023 Takes Responsibility
for Own LearningP=.0072 Higher Self-Regard P=.0079
Table 7.
[Special Education Students Intercept P=.0001]
Results of Multiple Linear Regression on Special Education Students Variables
P
Male/Female P=.0001 Improved Content
Knowledge and SkillsP=.0146 Improved Critical Thinking
and Problem SolvingP=.0035 Increased Interest
in the Subject AreaP=.0156 Takes Responsibility
for Own LearningP=.0012 Greater Confidence as Learner P=.0006
