This article reports a documentary analysis of the validity of summative examination assessment tasks of nine English major courses of a Saudi pubic university. In order to make valid inferences from the assessment of students’ learning outcomes, it is imperative that assessment instruments are valid and reliable. Valid and reliable assessment of students’ learning outcomes is necessary for institutions, learners and society alike. It has been observed that most Saudi students get grades that are untrue of their actual academic achievements. One of the reasons behind this problem is a mismatch between CLOs and the assessment tasks. Therefore, this study evaluated the nature of end-of-course examinations given to English major students as a major part of the assessment. Thus, this study though very limited in scope carries immense significance to highlight assessment methods that obstruct the fulfillment of higher-order learning.

The Validity of Assessment Tasks

Valid assessment tools measure exactly what they are required to measure (Coombe and Evans, 2005) whereas invalid tests usually concede false inferences regarding students' achievement. Therefore, it is imperative for a language assessment task to show the actual achievement of learners otherwise the results are deceptive. Learners get commendable scores (certificates) without having achieved the intended learning outcomes (Green, 2007b). Therefore, Green (2007a) has given the model of overlap that asks for maximum overlap between exam specifications and the target academic and linguistic skills intended to be achieved (see Figure 1). The greater the overlap the greater is the achievement of learning outcomes.

A Model of Washback Direction (Green, 2007a)

Every educational program or course has certain learning outcomes that its learners are trained to achieve (Archbald and Newmann, 1988). To ensure the successful achievement of the learning outcomes, there is a need to examine the progress of learning. This aim can only be achieved by employing valid assessment tasks that are ‘worthwhile, significant, and meaningful—in short, authentic’ (Archbald and Newmann 1988, p.1). Such authentic assessment tasks require the production of new knowledge or applying concepts to novel situations rather than a mere reproduction of previously taught information (Muñoz and Álvarez, 2010). They have remarked that it is easy for teachers to decide if “students have a real understanding of the material presented and they are able to synthesize concepts” (2010, 44). However, the situation according to them can be different if the assessment tasks lack authenticity and have a strong similarity to teaching tasks. These causes make students memorize selected materials and resultantly teachers cannot determine the actual acquisition of the concepts and skills.

Previous Studies

A critical review of previous studies suggests that valid and authentic tests have mostly resulted in the achievement high-order learning (Muñoz and Álvarez, 2010; Benedetti, 2006; Saif, 2006; Ferman, 2004; Manjarrés, 2005; Stecher, Chun and Sheila, 2004; Cheng, 1997). For example, it has been noted that listening tests based on video are more reliable than their audio counterparts (Benedetti, 2006). The fusion of oral elements in end of school assessment had a beneficial impact on Matriculation students’ oral proficiency compared to pen and paper examination (Ferman, 2004). The study of Saif (2006) revealed that a strong alignment between test specifications and course learning outcomes generates observable positive washback. This was confirmed by Stecher, et al. (2004) in their study of reforms introduced to the assessment regime of the writing component of examinations in Washington State where changes in assessment tools showed a considerable positive impact on learning processes. In addition, the studies of Manjarrés (2005) and Muñoz and Álvarez (2010) also confirm the argument that a strong correlation between course learning outcomes and test specifications has a beneficial effect on both teaching and learning.

It has to be born in mind that the alignment itself may not yield expected results. Therefore, it is mandatory to apprise students regarding how to access assessment tasks. This argument is supported by the findings of Cheng (1997). It was found in the study that despite the assessment tasks being aligned with course learning outcomes students’ learning did not show the expected positive evidence. Therefore, teachers’ role becomes critical along with decision-makers or supervisors.

Invalid and unauthentic assessment tasks, however, are surely a major cause of lower-order and superficial learning as confirmed by empirical evidence in different settings (see for example Scouller, 1998; El-Ebyary, 2009; Gijbles, Segers and Struyf 2008; and Gijbels and Dochy, 2006). The findings of these studies show that students focus only on lexical and grammatical accuracy because assessment task designs are contrary to learning outcomes. Some assessment tasks even only measure lower order language skills (Gijbles, et al., 2008). Consequently, learners do not tend to focus on improving higher-order cognitive skills (Gijbels and Dochy, 2006). One example of such assessment tasks are multiple choice questions that mostly cause surface level learning (Scouller, 1998). From the two categories of studies with positive and negative washback, we can infer that strong congruence between assessment instruments and the outcomes of a course is indispensable else the entire exercise of education can pale into insignificance.

Based on the analysis of the tasks, two categories were formulated, i.e., SROs and CROs (see Table 1&2). The data analysis yielded very straight forward results. More than two-thirds of the tasks were SROs. The CROs were given mainly in literature exams. All linguistics and skill courses except Morphology were SRQs. Moreover, three-fourth of the total marks were allotted to the SRQs.

Table 1. Marks distribution for SROs

Skill Courses Assessment Tasks Mapped Against the CLOs

Coverage of CLOs by all assessment tasks of all domains

The study focused on three areas i.e., validity and authenticity of the assessment tasks and the level of consistency across different courses. The first research question sought to examine if the tasks assessed the intended learning outcomes. The findings indicated that three-fourth (68%) of the outcomes were not measured that referred to validity issues in the assessment regime across the whole program. Interestingly four of the courses (see Appendix A) did not assess any of the intended learning outcomes. The most interesting dimension of this point is the skill course assessment tasks. It is mandatory for such course examinations to assess the target skills. However, as the findings showed, apart from situational English none of the skill courses assessed any of the learning outcomes provided in the approved course specifications. As noted in the literature review, for assessment tasks to be instrumental in assurance of quality learning they have to be an explicit alignment with the course learning outcomes because assessment is not separate from teaching and learning processes; they are intimate and linked together (Frankland, 2007; Boud, Cohen & Sampson, 2001). Learners are stakeholders in assessment practices (Saville and Hawkey, 2004). Their perception of a test shapes their learning styles. Assessment tasks not covering course learning outcomes will concede superficial learning (El-Ebyary, 2009; Gijbles, Segers and Struyf, 2008; Gijbels and Dochy, 2006; Scouller, 1998). On the other hand, empirical evidence indicates modifying test specifications influence teaching and learning (Saif, 2006; Ferman, 2004). Therefore, based on the findings of previous research it is safely posited that the assessment task designing in the context of this study needs alignment with course learning outcomes to develop higher-order learning.

The second research question required an evaluation of the authenticity of the assessment tasks and the anticipated validity of the inferences drawn from students’ actual performance on the given tasks. The authenticity factor requires assessment tasks to resemble real-life-like performance (Green, 2006; Messick, 1996, and Archbald and Newmann, 1988). The findings showed that none of the assessment tasks were truly authentic in nature. The learning outcomes of courses such as Situational English, IELTS and Paragraph Writing should be assessed through tasks that can reflect students’ achievement of the skills instead of using SRQs tests which only show students’ knowledge. Thus higher-order cognitive, as well as interpersonal skills, remain untested. Therefore, any inferences drawn from students’ results on such a test cannot be reliable. Similarly, the CRQs do not make learners think critically, analyze, synthesis and evaluate information. The CRQs given in examinations seemed to require the reproduction of limited and memorized information.

Regarding the last research question, it was noted that almost three-fourth of the assessment tasks were SRQs. Five of the courses did not have any CRQs. In addition, the findings revealed vivid inconsistency in marks allocation and the space provided to be used by students for answering CRQs. For example, as shown in Tables 3 and 4, marks were allocated for two-line answers whereas five-line answers carried 15 marks. It should be made clear at this point that the students seemed to have an implicit but very strong message that they should use the allotted space only, i.e., no need for using extra space. Last but not the least data analysis revealed that most of the CRQs were short answers memorized by students with great resemblance to answers across whole classes. However, at the same time, it was found that all instructors-cum-examiners had an identical approach to assessment task designing. Almost all of them design SRQs and short answer questions. This may be due to an undeclared departmental policy towards assessment task designing or the teachers’ shared belief of assessment culture in the given context. This is evident from the dominance of SRQs in most of the tasks. It cannot be a mere coincidence that instructors with Ph.D. degrees as well MA qualifications design almost similar tasks. Previous research has also shown that teachers’ assessment task designing practices are often dictated by different stakeholders such as students, parents and community (Cheng, 1997). Therefore, further investigation may reveal the reasons for this undesirable assessment approach.

Teachers’ perception plays a critical role in their approach to teaching. Cheng (2002) has remarked that teachers’ perceptions concerning teaching emerge from various areas such as their personal learning and teaching experiences along with the kind of education system they have been students of. Their beliefs, values and objectives with regard to the syllabus and teaching methods are influenced by their perceptions experiences. Such perceptions also affect their concept of the curriculum they teach and their specific roles within it. Finally, perceptions influence their decision making and actions. The most significant factor forming teachers’ approach to teaching and learning is their familiarity with syllabus, skills and content assessed in a test (Alderson and Wall, 1993). The greater the knowhow, the more is the influence on pedagogic practices. Therefore, desirable alignment between teachers’ perceptions and the intended learning outcomes is indispensable and if not assessment literate already, they need to have adequate assessment literacy.

This research was based on the analysis of only nine courses (total number of courses was 36) of an English-major program. Its findings, therefore, may not be generalizable unless the assessment tasks of the whole program are evaluated. However, in the context of assessment tasks mapping with CLOs, this study has cast serious doubts on the validity of assessment tasks that can guarantee quality assurance required by NQF standards. The assessment tasks evaluated were completely invalid. They did not measure what they were supposed to measure. Evidence from previous research indicates that assessment tasks that are authentic and actually measure target CLOs have greater prospects of inculcating higher-order learning whereas assessment instruments with SRQs result in surface-level learning. Therefore, it can be safely stated that the English major summative examination is very far from promoting higher-order learning. Further research into the alignment between CLOs and assessment tasks of English major programs from another Saudi university will show the extent this problem contributes to the low quality of students’ learning nationwide. Experimental studies, in particular, may show how assessment tasks having greater congruence with CLOs can ascertain quality assurance required by NCAA/NQF. Future research may seek teachers (assessment task designers) input to determine the factors contributing to their current assessment practices and literacy.