Methodology

Sample

Data from online and face-to-face classes from both summer and fall semesters were analyzed.  Students were junior and senior business majors at a large state university.  Partial or complete information was collected from 52 students.  Table 1 presents the numbers of students in each class that started and participated in at least one assignment.

Table 1   Number of students in each class

Assignments used as data

Assignments collected for analysis included the following: 

1) Cover letter

Standard cover letter for a resume.  This was the first assignment given during the semester and is used as a measure of student writing ability (without instruction) at the beginning of the class.

2) Ethical dilemma memo

Students analyzed an ethical dilemma and then advocated a correct course of action.

3) Analytical essay

Students wrote a six-page (1200 word) essay about a current business topic.  Students presented a thesis and supported the thesis with evidence.  This assignment was not collected from the summer class because of logistical reasons.

4) Final project

Student wrote a persuasive essay with a thesis and supporting evidence.  Final project expanded on the analytical essay by including more detail and by addressing opposing viewpoints to the main thesis.

Rating procedure

A five-point rubric evaluated students on four main criteria: organization, content,  mechanics, and overall clarity.   The four criteria are referred to as “writing dimensions” in this analysis.  The summation of all sub-scores was also used as a total score.  Table 2 shows the rubric used for grading.

Table 2  Rubric for independent grader

The independent rater designed the rubric after reviewing the instructor’s grading criteria.  The rater did not know the class identification of students (OL vs. FTF).   Additionally, any potential differences in format that could have been used to identify the type of class were removed or changed by the researchers.

Total score ratings for each assignment were compared with the teacher’s grades.  Overall, the correlation between grades of the teacher and the independent rater’s scores were high, one indicator of the reliability of the ratings.  Table 3 presents Spearman-Brown correlations for each assignment.

Table 3  Correlations between teacher’s grades and independent rater

Internal reliability of the ratings was comparable with Spearman-Brown correlations.  The reliability estimate used the 16 separate measures across the four assignments.  The Cronbach’s Alpha internal reliability coefficient equaled .73.

 Composite measures of each writing dimension correlated moderately with each other with correlations among the composites ranging from .4 to .6.  The pattern of correlations suggests that the writing dimensions tap some independent variance.  An exception to this pattern was the MECHANICS composite variable which exhibited a low correlation (.13) with the ORGANIZATION composite variable.

Results

The dependent variables used in the analysis were the separate writing dimension scores from each writing assignment as well as composite variables that averaged scores across each writing dimension.  Table 4 presents means and standard deviations for all measures by CLASSTYPE and CLASS.

Table 4  Means and Standard Deviations for each dependent variable by CLASSTYPE and CLASS

The first analysis compared the mean ratings for each separate rubric dimension for each assignment that reflected instruction.   Analyses used multi-factorial ANOVA with fixed factors CLASSTYPE (OL/FTF) and SEMESTER (fall/summer).   Additionally, because differences in mean scores among the four classes could balance each other and hide individual differences among classes (the CLASS variable), a Tukey post-hoc test was conducted to learn if individual class means differed from each other.   Table 5 summarizes the results from the ANOVA’s and the Tukey posthoc tests for each measure. 

Alpha level for significance is set at p < .05, with total degrees of freedom equal to 46 - 51. 

*    a < .05

**  a < .01

Table 5   ANOVA tests for each outcome measure

Progress from the beginning to end of the course was evaluated with pre- and post measures.  Measures along each writing dimension compared the first writing assignment (the cover letter reflecting no instruction) with the final project.   A repeated-measures ANOVA was used to examine possible mean differences and interactive effects for CLASS and CLASSTYPE.  Differential progress among the four classes was evaluated with Tukey posthoc tests.  Table 6 summarizes the results from the repeated measures ANOVA’s.

Alpha level for significance is set at p < .05, with total df = 44  

*    a < .05

**  a < .01

Table 6 Repeated-measures comparisons for cover letter/final project (Pre/Post)

 Because the fall FTF class had an anomalously high score for certain writing dimensions on the cover letter assignment, the same repeated-measures analysis was conducted treating the fall FTF class as an outlier.  Table 6.5 summarizes the results from the repeated-measures ANOVA with the fall FTF class removed from the analysis.

Table 6.5 Repeated-measures comparisons for cover letter/final project (Pre/Post) [NO FALLFTF]

Composite variables were formed for each writing dimension by summing each writing dimension score for each assignment as well as the total score.  Five composite variables were formed: ORG (organization), CON (content/ideas), MECH (mechanics), CLARITY (overall clarity) and TOTAL.  Only assignments that reflected instruction were used, and averages included as much (or as little) information as was available for each student in each dimension.  Figures 1 and 2 graphically present the means for CLASSTYPE and CLASS across each writing dimension.  Table 7 summarizes univariate ANOVA comparisons for each dependent variable for CLASSTYPE and CLASS.

Table 7  ANOVA results for composite variables

Alpha level for significance is set at p < .05, with total degrees of freedom equal to 51. 

*    a < .05

**  a < .01

Analysis of possible self selection

To gauge the amount of possible self-selection into class format a survey administered at the beginning of the semester asked students to respond to questions about attitudes toward computers, computer usage and computer training.  The survey contained 31 questions and composite variables were formed from groups of questions identified through factor analysis as representing common variance. No significant differences were found in any of the composite variables related to selection.   Table 8 summarizes t-test results for composite selection variables for the summer semester.

Table 8  t-tests of selection variables

Discussion

The current study seeks to avoid some potential weaknesses of distance education comparisons.  First, students from more than one class were sampled to overcome a lack of power due to numbers.  This effort was only partly successful with a total group size of 52.  Additionally, sampling from summer and fall semesters examined if online and face-to-face comparisons could be generalized over semesters.  In this case, the differences were much greater for the summer class than fall, indicating an interaction between semester and class type.  Power was also gained by using a more detailed rubric than was available from the class instructor.  More detailed grading allowed different writing dimensions to be assessed.  The use of a rubric by an independent rater (who was blind to class type) also avoided any possible unconscious grading bias on the part of the instructor.

Examination of table 4 (mean comparisons for each measure) shows that for each assignment’s total score, students from the online class did slightly better than their peers in the face-to-face class.   The largest differences among individual class total score means occur during the summer, with much smaller differences, and differences favoring the face-to-face class occurring during the fall.  Differences in the total score measures, and for individual measures reach statistical significance only for the ethical dilemma assignment.  It should be noted that much of the statistical differences between CLASSTYPE are due largely to higher scores for the summer on-line class and lower scores for the fall face-to-face class.

For composite measures of each writing dimension the largest mean differences are shown in the CLARITY, CONTENT and ORGANIZATION variables, with almost no difference between type of class in the MECHANICS variable.  The mean differences across writing dimension are aggregated in the TOTAL variable which shows a statistically significant difference favoring the online class.  Again, most of the difference occurs for the summer students, although fall online students do slightly better on the CLARITY, CONTENT, and ORGANIZATION composites than the fall face-to-face students, but perform worse on the MECHANICS composite. 

The pre/post comparison made between the cover letter and the final project along each writing dimension is complicated by an abnormally high scores on the cover letter by the fall face-to-face students.  One hypothesis about this difference is that these students are better writers at the outset of the class.  However, if this were the case their writing skill disappeared as the semester continued.  A more plausible explanation for this result might be differential instruction.  Field notes indicate that students in the fall face-to-face class received instruction about the assignment while the other three classes did not receive instruction.  Pre/post comparisons(leaving out the fall ftf class) suggest that most of the writing improvement occurred in the ORGANIZATION and CLARITY writing dimensions.   Overall improvement along each dimension was aggregated in the total variable, which exhibited a significant pre/post difference.

Little evidence exists that self-selection occurred along any dimension related to computer use, attitudes, or training, or that one class or another had more or less experience writing at the beginning of the class.  Students from both classes also had comparable grade point averages, age, gender and marital status (none were married).   The aforementioned differences in the cover letter assignment probably do not indicate that the fall face-to-face group were better writers at the beginning of the class.  However, if these scores were taken at face value and used as a covariate adjustment mean differences due to CLASSTYPE actually become larger in favor of the online group.

A second group of analyses compared text between the online and face-to-face classes for three class days during the summer semester.   Two analyses were conducted.  First, proportions of teacher and student “talk” (writing for the online class) for each class format were tallied for the three-day class period.  The same analysis was conducted for class time devoted to talking about writing.  Secondly, general activities and talk about writing were analyzed to learn if there were differences in activities and content between class formats. 

Researchers used text lines from written transcripts of audio-taped face-to-face classes and printouts from archived bulletin-board dialogues from the online class as the unit of analysis.  Because of non-comparability in the overall number of lines between written and spoken text, proportions were used to compare classes instead of quantities.  Several difficulties arose in comparing talk between online and face-to-face classes for proportions of total class time devoted to specific activities.  For the writing class, several general activities could not be compared because of their asynchronous nature.  The most obvious difference between the two types of classes is the lack of synchronous lecture time for the online class.  Online students can read as much or as little as they want from the “writing tips” section of the website, while face-to-face students listen to the teacher lecture in class.  Additionally, some of the talk about the class activities including grading, deadlines, and materials is removed from class discussion in the online class as   students access separate sections of the web page for scheduling and deadlines while face-to-face students often discuss scheduling and assignments in class.  Finally, online students often email teachers individually with questions about class content.  While students in the face-to-face class can also email and visit the teacher individually, direct questions about content asked by students during a face-to-face class are often asked on email by the online students.  The one activity which was roughly analogous (and comparable) between classes was talk about writing.  Talk about writing in both types of classes focused upon writing topics, the content of what was written, clarification of thesis and supporting evidence, writing mechanics and wording.  

The first analysis focused upon the proportion of time the teacher and students talked in the face-to-face class compared with the same proportion for the online class.  For the three day time period, the proportion of teacher talk compared to student talk was dramatically different between classes.   Table x also presents the same comparison for talk about writing, the one class activity that was roughly analogous across both types of classes.   

-------  table x ---------------

While proportions of general class activities were not strictly comparable across formats, the comparison illustrates the proportion of teacher-student contact time spent talking about specific topics.  The four codes were used to classify topics are presented in table x + 1 with proportions of class time devoted to each activity.

Table x  Proportion of total class time, and talk about writing for teacher and students

The final analysis focused on talk about writing.  The main division used for analysis compared talk about content, ideas, and thesis as opposed to talk about grammar, spelling, format and word choice.  Codes were created for the two main types of talk.  Codes are included in table x + 2, with proportions for each code.

Discussion

Analyses show differences in the amount of text lines produced by the teacher and students in each class format.  While the greater presence of the teacher in the proportion of talk for the total class can partly be explained by the addition of lecture during the face-to-face class, the teacher also talks much more during class time devoted to talk about writing.  An obvious difference between face-to-face and online discussions about writing is the length of an average utterance.  In the face-to-face class the average utterance by the teacher and students during talk about writing was much shorter than the OL course (2.2 lines FTF compared with 8.4 lines OL).    During the FTF course the teacher interjected much more often and asked direct questions of the students.  In the OL course, students were much more likely to give a lengthier commentary on another student’s writing without much guidance or interjection from the teacher.

In table x + 2 the proportions of text devoted to content compared with mechanics by both teachers and students is roughly similar between class formats.   However, it should be noted that the total text devoted to content and mechanics is a weighted average given that the teacher’s presence is less online than in the face-to-face class.   The total amount of time spent devoted to content (versus mechanics) is slightly less in the online class.  This result directly contradicts the result found in figure 2 of the outcome section which shows OL students doing better in areas of content, organization and clarity, but performing in a similar manner on the mechanics composite.  For this reason, no connection can be made between the text analysis and the outcome section.  [Actually, we need to sample better from the texts if we want to make the connection that the proportion of time devoted to content/mechanics has anything to do with the outcome.  This means more transcription and coding….]