Comparative and non-comparative evaluation in educational technology

This post present two examples of comparative and non-comparative studies:

Comparative Study

1. Title: Comparative Analysis of Learner Satisfaction and Learning Outcomes in Online and Face to Face Learning Environment


2. Type of comparative study: learner's perception and performance


3. Problem: Because the sharp growth of online programs in recent years, there is a need for researches to assess the capabilities and efficacy of online programs.


4. Purpose of evaluation: compare an online course with an equivalent course taught in a traditional face-to face format.
5. Question:
   What differences exist between students enrolled in online versus face-to-face learning environments in: *satisfaction with the learning experience?
   *student perceptions of student/instructor interaction, course structure, and course support ?
   *learning outcomes (i.e., perceived content knowledge, quality of course projects, and final course grades)?
6. Participants: Students enrolled in instructional design course for human resource development professionals. *19 students: were taught on a traditional face-to-face format
   *19 students: were taught totally online.
7. Instruments applied
   1) some items from the Instructor and Course Evaluation System (ICES)
   2) Course Interaction, Structure, and Support (CISS) instrument
   3) A course projects
   4) The final course grades
   5) A self-assessment instrument:


8. Advantages of the study:
   1. Equivalence of the groups in number, same instructor and course, delivered by the same department, and required the same content, activities, and projects
   2. All data were collected at or near the end of the semester
   3.To ensure instrument validity, Researcher used many ways.
   a.To develop CISS the researcher contacted with authors of the (DOLES) and (DDE) instruments to obtain copies and necessary permission to use their instruments. b.Content experts reviewed the items of instrument.
   c.The instrument was pilot tested by 68 students
   d.Factor analysis procedures were used to establish the construct validity of the (CISS) instrument.


9. Disadvantagesof the study:
   o The small sample size makes it difficult to interpret the result.
   o The CISS instrument is still in its early developmental stage and has not completed a full analysis to ensure reliability and validity.


10. Results of the study:
    1.Student Satisfaction: on instructor quality and course quality, both groups provided positive ratings .
    2.Perception of course interaction, structure and support: Overall, both groups of students had positive perceptions. with the face-to-face students having significantly more positive views for interaction and support.
    3. Student Learning Outcomes:
    §Blind review of course projects: The difference in the project ratings for the two groups was not significant
    § Course grade: The grades were, for the most part, equally distributed between both groups
    § Self- assessment: Significant differences were found on only five of the 29 items on the self-assessment instrument
    

Note:At the end of the research, the researcher said: "These results support the argument that online instruction can be designed to be as effective as traditional face-to-face instruction"

11. Refernce:
JOHNSON, S. ARAGON,S. SHAIK,N. PALMARIVAS, N. (2000). Comparative Analysis of Learner Satisfaction and Learning Outcomes in Online and Face to Face Learning Environment. USA

Non-Comparative Study

1. Title: Virtual interactivity: design factors affecting student satisfaction and perceived learning in asynchronous online courses
2. Purpose: This study is a non-cooperative study. It investigates factors affecting student satisfaction with and perceived learning from asynchronous online learning.
3. Participants: Participants of this study are Approximately 3,800 students who were enrolled in 264 courses offered through SLN but About (1,406) students returned the survey. Also, about 73 courses and eleven hundred and eight (1,108) students were enrolled in the. courses whose design features we examined.
4. Evaluation instruments:
   This study is used online SLN Student Survey Data as a data collection of tool. It is consisted of mostly multiple choices, forced answer questions eliciting demographic information and information concerning students’ satisfaction, perceived learning, and activity in the courses they were taking. In addition, it has open-ended comments for respondents to add comments to the survey.
   Course Design Data: Two of the researchers separately examined each of the 73 courses and rat their content on twenty-two variables using Likert-type scaling. Ratings for each course were checked for agreement, and disagreements were resolved by consensus with reference to the courses themselves.
5. Advantages:
   *Used more than one instrument for collecting data.
   *Gives some information about why the finding is that.
6. Disadvantages:
   *No clear information about participants.
7. Results:
   It shows high levels of satisfaction with and perceived learning from SLN courses in the Spring, 1999 semester. The findings also indicate that most students believed their level of interaction with the course materials, with their instructor, and with their peers was as high or higher than in traditional face-to-face courses.
   The study comes with that, there are three factors that contributing significantly to the success of online courses. Those factors are a clear and consistent course structure, an instructor who interacts frequently and constructively with students, and a valued and dynamic discussion.
8. Refernce:

Swan, K. (2001). Virtual interactivity: design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22, (2), 306-331.