A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong

In the field of writing assessment, various factors have been identified to influence the validity of examinees' scores. One of the most prominent factors believed to threaten scoring validity in an assessment is the raters. Raters played an important role in any type of assessment particularly...

Full description

Bibliographic Details
Main Author:	Embong, Nurrul Che Eme
Format:	Thesis
Language:	English
Published:	2013
Subjects:	Terengganu Language
Online Access:	http://ir.uitm.edu.my/id/eprint/14622/ http://ir.uitm.edu.my/id/eprint/14622/1/TM_NURRUL%20CHE%20EME%20EMBONG%20ED%2013_5.pdf

id	uitm-14622
recordtype	eprints
spelling	uitm-146222016-09-05T03:53:24Z http://ir.uitm.edu.my/id/eprint/14622/ A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong Embong, Nurrul Che Eme Terengganu Language In the field of writing assessment, various factors have been identified to influence the validity of examinees' scores. One of the most prominent factors believed to threaten scoring validity in an assessment is the raters. Raters played an important role in any type of assessment particularly ones that involve writing. The raters need to be reliable in their rating ability and awarding marks which will be used to determine the examinees ability. Previous research has shown that the most concerned group of raters would be the novice raters. This study seeks to investigate the inter-rater reliability of novice raters in rating using holistic and analytic scoring rubrics for writing assessment, particularly in rating expository essays for BEL 120 course in UiTM Dungun, Terengganu. The three novice raters chosen for this study were selected based on the same characteristics. The raters were asked to rate 30 expository essays using Test of Written English (TWE) holistic scoring rubric first and after an interval of two days, they were asked to rate the same essays using BEL 120 analytic scoring rubric used by the faculty to mark BEL 120 final exam. The marks of the essays were computed and analyzed using SPSS Version 18.0 for Windows to generate the results. Intraclass Correlation Coefficient was used to determine interrater reliability since it involved more than two raters. The results showed that the novice raters have low inter-rater reliability level for both scoring rubrics used. The scores discrepancies among the raters also varied greatly thus the raters have low raters agreement when awarding marks using both scoring rubrics. Although the raters were familiar with analytic scoring rubric, the results showed that the inter-rater reliability level for analytic scoring is lower than holistic scoring. The findings provide an insight on the actual level of novice raters' inter-rater reliability and appropriate action is hoped to be taken by Academy of Language Studies in the institution to increase the raters' reliability in rating writing assessment as to increase scoring validity of the examinees. 2013 Thesis NonPeerReviewed text en http://ir.uitm.edu.my/id/eprint/14622/1/TM_NURRUL%20CHE%20EME%20EMBONG%20ED%2013_5.pdf Embong, Nurrul Che Eme (2013) A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong. Masters thesis, Universiti Teknologi MARA.
repository_type	Digital Repository
institution_category	Local University
institution	Universiti Teknologi MARA
building	UiTM Institutional Repository
collection	Online Access
language	English
topic	Terengganu Language
spellingShingle	Terengganu Language Embong, Nurrul Che Eme A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
description	In the field of writing assessment, various factors have been identified to influence the validity of examinees' scores. One of the most prominent factors believed to threaten scoring validity in an assessment is the raters. Raters played an important role in any type of assessment particularly ones that involve writing. The raters need to be reliable in their rating ability and awarding marks which will be used to determine the examinees ability. Previous research has shown that the most concerned group of raters would be the novice raters. This study seeks to investigate the inter-rater reliability of novice raters in rating using holistic and analytic scoring rubrics for writing assessment, particularly in rating expository essays for BEL 120 course in UiTM Dungun, Terengganu. The three novice raters chosen for this study were selected based on the same characteristics. The raters were asked to rate 30 expository essays using Test of Written English (TWE) holistic scoring rubric first and after an interval of two days, they were asked to rate the same essays using BEL 120 analytic scoring rubric used by the faculty to mark BEL 120 final exam. The marks of the essays were computed and analyzed using SPSS Version 18.0 for Windows to generate the results. Intraclass Correlation Coefficient was used to determine interrater reliability since it involved more than two raters. The results showed that the novice raters have low inter-rater reliability level for both scoring rubrics used. The scores discrepancies among the raters also varied greatly thus the raters have low raters agreement when awarding marks using both scoring rubrics. Although the raters were familiar with analytic scoring rubric, the results showed that the inter-rater reliability level for analytic scoring is lower than holistic scoring. The findings provide an insight on the actual level of novice raters' inter-rater reliability and appropriate action is hoped to be taken by Academy of Language Studies in the institution to increase the raters' reliability in rating writing assessment as to increase scoring validity of the examinees.
format	Thesis
author	Embong, Nurrul Che Eme
author_facet	Embong, Nurrul Che Eme
author_sort	Embong, Nurrul Che Eme
title	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_short	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_full	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_fullStr	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_full_unstemmed	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_sort	study of inter-rater reliability among novice raters of the bel120 writing course in a university in malaysia / nurrul che eme embong
publishDate	2013
url	http://ir.uitm.edu.my/id/eprint/14622/ http://ir.uitm.edu.my/id/eprint/14622/1/TM_NURRUL%20CHE%20EME%20EMBONG%20ED%2013_5.pdf
first_indexed	2023-09-18T22:52:03Z
last_indexed	2023-09-18T22:52:03Z
_version_	1777417609052946432

A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong

Similar Items