English Study Design


Home

Project Team

1999 Project Team Report

The English Study Design in Detail

 A Guide to Planning and Assessment

Cedric Hall: Report on the 1998 year 12 trial

 Terry Locke: "Assessment Standards: What's in a Name?"

 The 1999 Reference Test Trial

 Achievement 2001: Update and Forum

Contact Us

 

EVALUATION OF THE YEAR 12

ENGLISH STUDY DESIGN TRIAL

1998

Report prepared by:

Cedric Hall
Head of School, School of Education
Victoria University of Wellington
PO Box 600
Wellington


Report to Dr Terry Locke
English Study Design Co-ordinator
February 1999


 Background to the evaluation

 Interim evaluation

 Current evaluation

 Results of the HOD questionnaire

 Results of the teacher questionnaire

 Results of the student questionnaire

 Summary

Background to the evaluation

This report sets out the reactions of Schools to the Year 12 English Study Design (ESD). Thirteen schools from three cluster areas -- Auckland, Christchurch and Otago/Southland-- elected to participate in the trial.

The ESD is a standards-based programme for sixth form (Year 12) English. Underpinning the ESD are the processing strands of the national curriculum in English: critical thinking, exploring language and processing information. These are not discrete divisions of the programme but related elements which are blended through the course activities undertaken by students.

The ESD translates the national curriculum into a set of course components, each with its own weighting. These are:

 WORK REQUIREMENTS

 WEIGHTING

Workfile
Writing Folio
Response to Text
Language Investigation Project
Communication Project

0.10
0.24
0.35
0.15
0.15

During preliminary cluster meetings in February 1998, participating schools indicated a desire to provide a standards-based sixth form English programme for their students which was an alternative to unit standards. All schools saw little difficulty in adapting all or most of their existing programmes to the ESD. Most schools adopted the full design but a few omitted either one or two of the requirements or substituted an alternative requirement.

Interim Evaluation

An interim evaluation was conducted in July 1998 drawing upon the initial experiences of schools to the ESD (Hall, 1998). The general reaction was that teachers and students had few problems in adapting to the ESD because it was close to what was already offered. Several schools made the comment that students would not be aware that the programme was in any way different from usual. Staff were generally positive about the content, structure, manageability, workload, method of assessment and student-centred focus of the ESD. However, some schools experienced difficulties applying the assessment and some students still did not cope well with aspects of the programme, although it is hard to know whether such problems were less to do with the programme and more to do with usual variations between students in their knowledge and skills.

One school commented that if the ESD was offered as an official alternative to the current (unit standards) system, it would be embraced with enthusiasm. Another school commented that complications had arisen because a unit standard in transactional writing was being offered at the same time and that handling different systems was complex and hard to manage. This school found that the unit standard format was not well adapted to such writing. The same school noted that the ESD was working very well in respect of the writing folio -- high quality work was being produced. Another school noted that it was offering two unit standards and found these inflexible in comparison with the components of the ESD being taught.

 

Current evaluation

The evaluation was conducted towards the end of the school year and comprised three questionnaires, one each for the HOD, teachers of the programme and students.

Each questionnaire was developed during cluster meetings with schools in July 1998 and is based on themes deemed important for assessing whether the programme was working well. During cluster meetings, draft questionnaires were developed and then refined through comment from those present.

Towards the end of October, the complete set of questionnaires was sent to each school. Unfortunately, only ten of the 13 participating schools were able to undertake the survey, and two of these omitted the student evaluation. One of the schools which did not supply data contacted the evaluator to apologise, identifying unforeseen factors which had limited the opportunity to collect the data (the temporary employment of replacement staff and students being involved in rehearsals for an end-of-year production). One of the two schools which omitted the student evaluation component ran into timetabling problems at the end of year, while the other did not sight the evaluation material until it was too late to administer the student component.

Results of the HOD questionnaire

The HOD questionnaire asked only one question:

Your school has elected to adopt the English Study Design, or aspects of it, for its Sixth Form Certificate English programme in 1998. What are the key factors that contributed to your school's decision?

All but one school commented on the need to find an alternative to the unit standards system, and seven schools commented negatively on the norm-based Sixth Form Certificate grade system. The results confirmed the interim evaluation findings which identified a need for a standards-based system of assessment but which did not involve the use of unit standards. The following quotations summarise the general response of schools:

Strong disapproval of Unit Stds which we see as fragmented and invalid as a way of assessing English.

Concern that unit standards methodology is unproven, poorly moderated, time consuming, difficult for students to understand, and that unit standards limit scope for broad range of teaching experiences.

As a department we were very concerned about "unit standards" as a method of teaching and assessment. Overall there was a philosophical opposition to unit standards. We also had a strong desire to embrace a system of assessment that did not tie 6FC grades with S-C results (as the present system does).

The ESD allowed us to escape the shackles of 6FC without embracing the flawed unit standards approach. It enabled us to be positive and forward thinking thereby avoiding being categorised as reluctant to change or afraid of change.

It offered a sensible end to the present shortcomings of the 5th and 7th Form exam system (i.e. the insanity of testing production activities/speaking etc. in exams) while avoiding the aimless multiplicity of unit standards.

It should be noted that although some of the above quotations identify avoidance of the 6FC grading system as a reason for adopting the ESD, the final standards-based grades obtained by students still required transformation to 6th Form Certificate grades at the end of the course. All schools recognised this. The trial, however, provided an opportunity for another system of grading to be investigated.

A second major reason given for adopting the ESD was that it represented a natural development of existing programmes.

Desire to keep the best aspects of our F6 course, which ESD has allowed. Keep the creative programme -- which unit standards would ruin.

We believed that there was little wrong with teaching delivery or what we were teaching. Sure it needs constant adapting but not "throwing out" as unit standards appeared to advocate.

The ESD allowed us to retain what was good, adapt where necessary and still feel that we were relating to the needs of students in a manner we could manage.

We were attracted by the Study design because we could see that it grew out of what we were already doing.

Other reasons given for adopting the ESD included its apparent manageability for teachers and students, the use of a simple external moderation system, coherence in programme content and structure, and a manageable assessment system.

 

Results of the teacher questionnaire

The teacher questionnaire focused on a number of themes which the cluster meetings deemed important for inclusion in the evaluation. The themes covered:

A. Scope and coherence of the ESD (2 items + comments)
B. Workload and manageability (3 items + comments)
C. Flexibility (3 items + comments)
D. Assessment (6 items + comments)
E. Moderation (2 items + comments)

In addition, the first item of the questionnaire asked each teacher to identify the components of the ESD that were included in their 6FC programme, and the last two items asked teachers to identify the three things they most liked about the programme (item 18) and the changes they would like to see made to the ESD (item 19). In all, 17 teachers completed the questionnaire although in some cases items were omitted when the teacher was unsure of her/his answer or was not in a position to provide a response.

Components included in the ESD

Five of the ten schools that responded to the questionnaire included all five components of the ESD in their 6FC programme. Three schools included four components, one school included two components and one school included only one component. In respect of the five components, item 1 identifies the number of schools which studied each.

Workfile
Writing Folio
Response to Text
Language Investigation Project
Communication Project
9 out of 10 schools
7 out of 10 schools
9 out of 10 schools
7 out of 10 schools
8 out of 10 schools

Item 1: Number of schools trialing each work requirement/component of the ESD

A. Scope and coherence of the ESD

Items 2 and 3 identify the questions asked of teachers in respect of the scope and coherence of the ESD. Teachers were asked to rate items using the scale: strongly agree (SA); agree (A); neutral (N); disagree (D); and strongly disagree (SD). Below each item, indices for 'satisfaction' (SA + A) and 'dissatisfaction' (SD + D) are provided.

 

 Number

 Frequency

 Strongly Agree
 8

 47%

 Agree
 9

 53%

 Neutral
 0

 0%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 100%

 Index of dissatisfaction

 0%

Item 2: The scope of the ESD meets the requirements of the national curriculum

 

 Number

 Frequency

 Strongly Agree
 9

 53%

 Agree
 8

 47%

 Neutral
 0

 0%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 100%

 Index of dissatisfaction

 0%

Item 3: The ESD provides students with a coherent programme in year 12 English

Both items indicate a high level of satisfaction with the ESD amongst the participating teachers. Comments included:

I found I had to make only minor changes to the programme already running to comply with the new requirements of the ESD.

We wished to have more formal speaking, but otherwise were satisfied with the scope.

Difficult to achieve coherency in this part of the course (Workfile).

Well planned and well structured.

Needs more time for students to complete it than we currently have. There will be plenty of time if we go entirely onto ESD though.

Reflects current teaching practice well. V(ery) coherent methodology, easy to understand and explain.

B. Workloads and manageability

Items 4-6 identify the questions asked of teachers in respect of workloads and manageability. Again all items conform to the 'strongly agree' to 'strongly disagree' format.

All three items indicate a high level of satisfaction with the ESD amongst the participating teachers. The one school which reported a negative response for the manageability and workload of students was combining elements from two programmes -- the workload for the students was thus higher than that for other schools.

 

 Number

 Frequency

 Strongly Agree
 4

 25%

 Agree
 12

 75%

 Neutral
 0

 0%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 100%

 Index of dissatisfaction

 0%

Item 4: The ESD is manageable in terms of the workload of teachers

 

 Number

 Frequency

 Strongly Agree
 5

 31%

 Agree
 10

 63%

 Neutral
 0

 0%

 Disagree
 1

 6%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 94%

 Index of dissatisfaction

 6%

Item 5: The ESD is manageable in terms of the workload of students

 

 Number

 Frequency

 Strongly Agree
 8

 50%

 Agree
 8

 50%

 Neutral
 0

 0%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 100%

 Index of dissatisfaction

 0%

Item 6: The ESD is manageable in respect of record-keeping and other administrative requirements.

Comments included:

Although I agree with the above statements, I would like to add that I felt we were rushed to complete the content of the course. A number of students also commented similarly. They often made comments that it would've been better to do these things with more thoroughness. I have to concur here. The course is also a huge step up in difficulty from SC course and students need time to adjust to this.

Students will need to change. Currently they are used to many short term goals and some do not organise well for long term goals. We will have to learn how to help them on this.

Simple, accurate and easy to interpret.

Record-keeping was really streamlined.

Manageable and satisfactory workload.

The workload appears to me to be less that what I expected from students completing 6FC. I have also found my workload less in terms of marking requirements.

Really pleased with this. Of course the first time round was busy for us, but vastly better than unit standards. Working from generic criteria to our detailed expectations took time.

As a result of the rubric marking system I was able to place all comments, rubric and marks onto a computer programme and give students a comprehensive printout as well as being able to retain a copy for the dept. records.

C. Flexibility

Items 7-9 identify the questions asked of teachers in respect of flexibility.

While all three items indicate a high level of satisfaction with the ESD, it is apparent from the comments provided by some teachers (below) that not all components of the course were as flexible as hoped for, and that some aspects of the previous 6FC programme were seen as more flexible. However, it should be noted that although the index of satisfaction is lower overall for this section than sections A and B, no teacher rated any item negatively. Comments were provided by only one of the teachers who strongly agreed with each of the items -- presumably the satisfaction level of these teachers with the ESD did not warrant further explanation.

 

 Number

 Frequency

 Strongly Agree
 6

 35%

 Agree
 9

 53%

 Neutral
 2

 12%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 88%

 Index of dissatisfaction

 0%

Item 7: The ESD allows you to plan a range of teaching and learning contexts and activities

 

 Number

 Frequency

 Strongly Agree
 5

 29%

 Agree
 9

 53%

 Neutral
 3

 18%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 82%

 Index of dissatisfaction

 0%

Item 8: The ESD provides sufficient scope for introducing innovations in content and delivery

 

 Number

 Frequency

 Strongly Agree
 8

 47%

 Agree
 8

 47%

 Neutral
 1

 6%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 94%

 Index of dissatisfaction

 0%

Item 9: The ESD enables students to develop and demonstrate a wide range of English skills and knowledge

Comments included:

At times I felt the content/context and conventions of language assessment criteria were narrowing the scope of the programme (compared to previous years). This is probably indicative of how they were to me. The more used you are to something the more flexible you can be. I also find that "working backwards" i.e. designing tasks/activities/ assessments to suit criteria in a set assessment schedule is more difficult than setting goals/objectives and designing assessment to suit.

Personal strengths can be developed. Creativity with activities -- sufficient scope.

We were able to accommodate most of our course unchanged

I do not believe that the ESD has allowed me to be more flexible compared with teaching 6FC. In 6FC each topic usually involved a project and a summative assessment. The project allowed for flexibility in what you did and how you did it. In ESD there is certainly flexibility in the range of summative assessments, but that is really the end of it.

We felt constrained by having to have the language investigation presented as a seminar. Would prefer to include s peaking assessment elsewhere. Also felt that the requirements for language were too much like F7, whereas out students needed a more introductory topic, which we did, to make it wider in scope.

D. Assessment

Items 10-15 identify the questions asked of teachers in respect of assessment.

 

 Number

 Frequency

 Strongly Agree
 3

 18%

 Agree
 14

 82%

 Neutral
 0

 0%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 100%

 Index of dissatisfaction

 0%

Item 10: The range of summative tasks covered by the ESD provides a valid measure of student achievement.

 

 Number

 Frequency

 Strongly Agree
 1

 6%

 Agree
 10

 63%

 Neutral
 3

 19%

 Disagree
 2

 12%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 69%

 Index of dissatisfaction

 12%

Item 11: The assessment weightings for the different components of the ESD are appropriately balanced.

 

 Number

 Frequency

 Strongly Agree
 5

 29%

 Agree
 8

 47%

 Neutral
 3

 18%

 Disagree
 1

 6%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 76%

 Index of dissatisfaction

 6%

Item 12: The standards based criteria used in the ESD are helpful for assessing student work.

 

 Number

 Frequency

 Strongly Agree
 6

 35%

 Agree
 9

 53%

 Neutral
 1

 6%

 Disagree
 1

 6%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 100%

 Index of dissatisfaction

 0%

Item 13: The grading scale (A+, A, ... E+, E) used in the ESD provides enough scope to discriminate between the different levels of performance of your students.

 Category

 Number

 Frequency

10-20 scale points
 2

 13%

10 scale points
 12

 75%

5 scale points
 2

 13%

Item 14: Ideally, how many scale points do you think is enough in order to grade student work? Please note that the current grade scale (A+, A, ... E+, E) comprises 10 scale points. __________

 Method

 Number

 Frequency

External examination

 1

 6%

Assessment against unit standards

 0

 0%

The current ESD

 11

 69%

Other: please specify*

 4

 25%

* All specified external exam + ESD

Item 15: If you had a choice, which method of assessment do you believe would most benefit student learning?

It is clear that there was a high level of satisfaction with the range of summative tasks covered by the ESD (Item 10: Index of satisfaction = 100%), however, there was some dissatisfaction with the weightings (Item 11: Index of dissatisfaction = 12%). In respect of item 11, it should also be noted that only one teacher rated the item as 'strongly agree'.

While it is clear that, on balance, most teachers found the standards based criteria helpful (item 12), a few teachers were less satisfied (see comments below). The application of the same criteria to different tasks appears to lie at the heart of the problem for a few teachers.

With two exceptions, teachers indicated satisfaction with the 10 point scale (items 13-14) although two of the 'positive' teachers felt that the scale might be extended, giving responses of 10-20. The two teachers who rated the item as 'neutral' or 'negative' felt that 5 points were sufficient.

Given a choice of assessment system for benefiting student learning, most teachers preferred the ESD (item 15). However, one teacher opted for assessment by external examination and four teachers opted for a combined external examination/ESD system. No teacher opted for unit standards.

Comments included:

    I think the ESD is clearly able to show students what they need to do to improve and a number of students at the end of August commented on this aspect.

    I found it very difficult for very good students to be awarded top marks i.e. interpretation and application of differences between A & B grades was well-nigh impossible.

    Good spread of marks. Easier to differentiate between a '6' and a '7'. Students could succeed by demonstrating different strengths -- accommodates diversity of response.

    The criteria, while maybe needing more fine tuning in some areas, were specific and helpful in my opinion.

    The ESD provides assessment for a wide scope of English activities. Standards-based criteria work well in most respects but will need to be revisited and revised on a regular basis.

    The Language Assessment involved a great deal more work than the allocated weighting -- it needed more. Things like the viewing test and biographical writing were relatively easy (for) 5%. The Workfile and Reading Log require a great deal of work for 10%. Some criteria need to be adapted/changed.

    My main concern is that this method of assessment leaves students very unprepared for Bursary, particularly in the area of essay writing skills. I did not find the standards based criteria helpful. They are far too detailed and the wording is meaningless in a number of areas. In the case of the criteria, parts would apply from different grades.

    Weightings: I agree with the weightings in general, but feel that the language investigation and communication project require larger single items of assessment that the rest and are inflexible in their requirements. The students feel that the Workfile is under-weighted, but I don't agree.

E. Moderation

Items 16-17 identify the questions asked of teachers in respect of moderation. It should be noted that three teachers were not familiar with the system of moderation used in the ESD and therefore did not answer this section.

 

 Number

 Frequency

 Strongly Agree
 4

 31%

 Agree
 5

 38%

 Neutral
 4

 31%

 Disagree
 0

 0%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 69%

 Index of dissatisfaction

 0%

Item 16: The current moderation procedures are helpful.

 

 Number

 Frequency

 Strongly Agree
 5

 36%

 Agree
 7

 50%

 Neutral
 1

 7%

 Disagree
 1

 7%

 Strongly Disagree
 0

 0%

 Index of satisfaction

 86%

 Index of dissatisfaction

 7%

Item 17: A moderation test or examination across schools would be a useful component of the moderation process.

Item 16 indicates that most teachers (69%) were satisfied with the ESD moderation procedures (the remaining teachers gave neutral ratings), but an even greater percentage of teachers (86%) indicated support for the introduction of a moderation test, although one teacher opposed this.

Comments included:

I found moderation very helpful particularly because I am the only teacher using ESD at this school. Comments were helpful and when judgments coincide it is reassuring. I found that the amount required to be submitted was "excellent". It wasn't so much that it became a chore and there seems little point in upping the amount if marking is on the whole in agreement. A moderation test is unnecessary if the above is happening. It's like having an exam and internal assessment! It also provides something else to fit into the programme.

It seems a desirable form of assessment -- especially in light of new qualifications (2001) outline.

A moderation test would provide strong support for the full range of students. I believe this is important because we are assessing all strands. This movement away from the written language strand has lowered standards of accuracy in this area and a moderation test would give a "benchmark" assessment of where our students sit in this area.

Our minor concern of a moderation test would be the different starting points of schools in the ESD (a flexible system which should remain) which could compromise achievements of some students. Also would teachers then teach to the moderation test or examination?

Item 18: the three things teachers most liked about the ESD

The positive factors most commonly identified about the ESD by teachers focused on notions of:

  • flexibility, adaptability and sensitivity;
  • variety and diversity;
  • manageability and workload;
  • content and structure;
  • marking schedules and criteria;
  • moderation with other schools;
  • the challenging nature of the programme; and
  • fit with the strands of the NCF.

Comments echoed those given in the interim evaluation report (Hall, 1998).

Item 19: the changes teachers would like to make to the ESD

Themes which repeated themselves in the comments of teachers included:

  • the need for revision/rewording of the assessment criteria;
  • the need for further developments in moderation (e.g. introduction of an external test); and
  • revision of the weightings for the components.

One teacher suggested that a 'coordinator' be appointed who would organise the sharing of good ideas between schools about assessment and other matters. One teacher stressed the need for teachers to learn how to help students with time management and planning. One teacher felt the need for greater guidance on 'typicalities', arguing that sometimes a student's work was quite disparate in different areas or on different tasks. Two teachers commented on the need to develop the programme further so that it fitted with the year 2001 proposals, and three teachers identified specific components which they would want changed or modified (these have all been covered in comments to previous sections of the questionnaire).

 

Results of the student questionnaire

The student questionnaire was developed with a view to students providing their teachers with feedback on how they had found the programme. The thinking behind the evaluation was that students have a perspective on their own learning which should aid schools in terms of their programme design and teaching. It is important to recognise that most students would not have experienced alternative year 12 programmes in English (e.g. unit standards) and therefore the results provide no comparative information. Strengths or weaknesses in the programme from the students' perspectives say nothing about the strengths and weaknesses of alternative programmes.

Participants at the July cluster meetings agreed that it would be useful to have a standardised questionnaire which all schools should use. This would enable schools to compare student reaction in their schools with student reaction overall and thus suggest whether aspects of the programme might, or might not, require further tuning within their school. Because of the effort needed in analysing data, it was agreed that the questionnaire would comprise mainly 'closed' format items. While this loses some of the richness that qualitative data provide, given the large number of students involved, no practical alternative was available at this point. It was agreed that future evaluations would draw upon the findings of the first year trial and modifications to the evaluation process would then be considered.

The student questionnaire comprised 19 items (17 closed, 2 open) based on general responses to the programme overall and particular responses to components of the programme. Not all schools answered every question -- where a component was not taught in a school the associated items were omitted.

General aspects of the programme

Items 1-4 focus respectively on course organisation, quality of course information, stimulation of interest and workload. Students were asked to rate items using 'strongly agree' to 'strongly disagree'. Below each item, indices for 'satisfaction' and 'dissatisfaction' are provided along with the range (highest and lowest values) across schools. This last piece of information gives an indication of the variability across schools in respect of students' satisfaction with aspects of the programme. A wide variation would suggest that some schools are either more effective in their teaching or they have developed approaches to aspects of the programme which students have found positive for their learning.

 

 Frequency

 Strongly Agree

 14%

 Agree

 39%

 Neutral

 31%

 Disagree

 13%

 Strongly Disagree

 3%

 Index of satisfaction

 53%

 Index of dissatisfaction

 16%

 Range of satisfaction

 82% -- 27%

 Range of dissatisfaction

 0% -- 49%

Item 1: I found the English programme this year to be well organised.

 

 Frequency

 Strongly Agree

 17%

 Agree

 37%

 Neutral

 31%

 Disagree

 14%

 Strongly Disagree

 1%

 Index of satisfaction

 54%

 Index of dissatisfaction

 15%

 Range of satisfaction

 73% -- 32%

 Range of dissatisfaction

 0% -- 39%

Item 2: Information about course content, assessment and other important matters has been clearly communicated in this course.

 

 Frequency

 Strongly Agree

 7%

 Agree

 25%

 Neutral

 45%

 Disagree

 18%

 Strongly Disagree

 5%

 Index of satisfaction

 32%

 Index of dissatisfaction

 23%

 Range of satisfaction

 56% -- 21%

 Range of dissatisfaction

 16% -- 33%

Item 3: Overall my interest in English has been stimulated by the content and presentation of this course.

 

 Frequency

 Strongly Agree

 11%

 Agree

 38%

 Neutral

 34%

 Disagree

 12%

 Strongly Disagree

 4%

 Index of satisfaction

 49%

 Index of dissatisfaction

 16%

 Range of satisfaction

 67% -- 24%

 Range of dissatisfaction

 4% -- 31%

Item 4: I found the workload for this course manageable.

It is evident from items 1-4 (and later items also -- see below) that the general level of satisfaction of students with the programme is less than the satisfaction levels noted earlier from the teachers' questionnaire. Although few items are directly comparable between the two questionnaires, it is evident that students are less favourably disposed to the programme than teachers. This, of course, is hardly surprising and likely to be a finding for any programme -- teachers are subject specialists, students are learners. It should also be noted that students are required to study Year 12 English -- they do not have a choice as they do for other subjects.

As a rule-of-thumb, it would be desirable that the index of satisfaction is at least twice that of the index of dissatisfaction. This is the case for items 1 (organisation), 2 (quality of information) and 4 (manageability of workload) but not item 3 (stimulation of interest). It is of interest that for all four items, a large percentage of students responded neutrally to these features of the course. Of greater significance are the range of satisfaction and range of dissatisfaction values obtained across schools: these suggest a very strong school/teacher factor in providing a course which students respond to favourably.

Basic skills

Items 5-6 focus on students' perceptions of their own development in respect of the basic skills of writing (item 5) and reading, viewing and listening (item 6). The results are positive in the sense that the index of satisfaction well exceeds the index of dissatisfaction (over three times the value for each item), however, it is a little worrying that some students felt that they had achieved very little over the year's programme. These results suggest that teachers should reflect on whether strategies for student self-assessment might be given a higher profile in the course -- the ability to self-assess one's strengths and weaknesses is an important component of learning.

 

 Frequency

 Strongly Agree

 13%

 Agree

 38%

 Neutral

 34%

 Disagree

 12%

 Strongly Disagree

 4%

 Index of satisfaction

 51%

 Index of dissatisfaction

 16%

 Range of satisfaction

 63% -- 37%

 Range of dissatisfaction

 11% -- 25%

Item 5: During this course my writing skills have improved.

 

 Frequency

 Strongly Agree

 10%

 Agree

 40%

 Neutral

 37%

 Disagree

 9%

 Strongly Disagree

 4%

 Index of satisfaction

 50%

 Index of dissatisfaction

 13%

 Range of satisfaction

 62% -- 33%

 Range of dissatisfaction

 0% -- 25%

Item 6: During this course my reading, viewing and listening skills have improved.

Assessment

Items 7-11 focus on students' perceptions of the assessment of their work. It should be noted that item 7 is on a different scale from other items -- a rating of 'about right' is the ideal value from a student's perspective. The index of satisfaction is therefore taken to be the percentage corresponding to a rating of 'about right'; the index of dissatisfaction in this case has been defined by adding together the percentages for the extreme responses ('far too many' and 'far too few'). There is, of course, a degree of arbitrariness in these categorisations.

 

 Frequency

Far too many

 6%

Too many

 28%

About right

 63%

Too few

 1%

Far too few

 1%

 Index of satisfaction

 63%

 Index of dissatisfaction

 7%

 Range of satisfaction

 89% -- 36%

 Range of dissatisfaction

 0% -- 17%

Item 7: The number of assessment tasks in this course was:

 

 Frequency

 Strongly Agree

 17%

 Agree

 42%

 Neutral

 32%

 Disagree

 8%

 Strongly Disagree

 1%

 Index of satisfaction

 59%

 Index of dissatisfaction

 9%

 Range of satisfaction

 67% -- 41%

 Range of dissatisfaction

 0% -- 21%

Item 8: The assessment tasks tested a wide range of my skills and knowledge.

 

 Frequency

 Strongly Agree

 16%

 Agree

 41%

 Neutral

 32%

 Disagree

 8%

 Strongly Disagree

 3%

 Index of satisfaction

 57%

 Index of dissatisfaction

 11%

 Range of satisfaction

 67% -- 41%

 Range of dissatisfaction

 0% -- 36%

Item 9: I usually knew what I had to do to get a good grade.

 

 Frequency

 Strongly Agree

 21%

 Agree

 33%

 Neutral

 28%

 Disagree

 15%

 Strongly Disagree

 3%

 Index of satisfaction

 54%

 Index of dissatisfaction

 18%

 Range of satisfaction

 86% -- 41%

 Range of dissatisfaction

 0% -- 36%

Item 10: I usually knew why I got the mark or grade I got.

 

 Frequency

 Strongly Agree

 19%

 Agree

 30%

 Neutral

 33%

 Disagree

 14%

 Strongly Disagree

 4%

 Index of satisfaction

 49%

 Index of dissatisfaction

 18%

 Range of satisfaction

 77% -- 38%

 Range of dissatisfaction

 5% -- 25%

Item 11: Comments and feedback on my work have been helpful.

The results are again positive in the sense that the index of satisfaction well exceeds the index of dissatisfaction for each item. Given that this was the first experience most schools had had with standards-based assessment at this level, the results are clearly promising. In this respect, items 9-11 get to the heart of standards-based assessment -- the communication of what is expected of students and feedback which provides the basis for students to improve their work. However, the range of satisfaction and dissatisfaction indices for these items indicates that not all schools have developed effective strategies for communication and feedback and will need to reflect on their current practices. In making this comment, the evaluator recognises that the communication of a standard is not something which can be done in words alone (e.g. by a statement of assessment criteria); standards are learned and internalised through practice and feedback.

ESD components

Items 12-17 focus on students' perceptions of the effectiveness of the Workfile, the Language Investigation Project, and the Communication Project for fostering the development of particular skills.

It is very clear that of the three components, the Workfile has been the least successful in achieving key objectives associated with it. In respect of organisational skills (item 12), the Workfile was viewed negatively by as many students as it was positively (29% each for the indices of satisfaction and dissatisfaction). In respect of helping students think about their progress (item 13), the results for the Workfile are even less satisfactory (27% for the satisfaction rating and 33% for the dissatisfaction rating). This item was the only one in the questionnaire which achieved a higher negative than positive rating. Clearly, more thought needs to go into the use of the Workfile for achieving its aims. However, the range of values across schools suggests that some schools are far more effective in using the Workfile than others. This further suggests that the sharing of ideas across schools might be a useful strategy to heighten the awareness of students about the purpose and value of the Workfile.

The results for both the Language Investigation Project and the Communication Project are more acceptable. The satisfaction ratings exceed the dissatisfaction ratings, although the figures still suggest that not all students are convinced about the value of these components. Again, the sharing of ideas and practice across schools might be a useful strategy for improving student learning.

 

 Frequency

 Strongly Agree

 8%

 Agree

 21%

 Neutral

 42%

 Disagree

 19%

 Strongly Disagree

 10%

 Index of satisfaction

 29%

 Index of dissatisfaction

 29%

 Range of satisfaction

 63% -- 18%

 Range of dissatisfaction

 16% -- 39%

Item 12: The Workfile helped develop my organisational skills.

 

 Frequency

 Strongly Agree

 6%

 Agree

 21%

 Neutral

 40%

 Disagree

 24%

 Strongly Disagree

 0%

 Index of satisfaction

 27%

 Index of dissatisfaction

 33%

 Range of satisfaction

 39% -- 18%

 Range of dissatisfaction

 16% -- 47%

Item 13: The Workfile helped me think about my progress in English.

 

 Frequency

 Strongly Agree

 9%

 Agree

 36%

 Neutral

 35%

 Disagree

 15%

 Strongly Disagree

 4%

 Index of satisfaction

 45%

 Index of dissatisfaction

 19%

 Range of satisfaction

 68% -- 37%

 Range of dissatisfaction

 12% -- 32%

Item 14: The Language Investigation Project has developed my understanding of the way language works.

 

 Frequency

 Strongly Agree

 9%

 Agree

 34%

 Neutral

 39%

 Disagree

 14%

 Strongly Disagree

 4%

 Index of satisfaction

 43%

 Index of dissatisfaction

 18%

 Range of satisfaction

 53% -- 34%

 Range of dissatisfaction

 5% -- 24%

Item 15: The Communication Project has increased my confidence in communicating with an audience.

 

 Frequency

 Strongly Agree

 13%

 Agree

 40%

 Neutral

 34%

 Disagree

 10%

 Strongly Disagree

 5%

 Index of satisfaction

 43%

 Index of dissatisfaction

 18%

 Range of satisfaction

 69% -- 39%

 Range of dissatisfaction

 10% -- 19%

Item 16: The Communication Project has increased my skills in preparing a presentation.

 

 Frequency

 Strongly Agree

 13%

 Agree

 41%

 Neutral

 36%

 Disagree

 8%

 Strongly Disagree

 3%

 Index of satisfaction

 54%

 Index of dissatisfaction

 11%

 Range of satisfaction

 81% -- 15%

 Range of dissatisfaction

 0% -- 32%

Item 17: The Communication Project has improved my ability to work as a member of a team.

Open ended questions

The two open-ended questions asked students to identify the components which they thought had be given too much weight (item 18) and the components which they thought had been given too little weight (item 19). The three components which received most attention were the Workfile, the Language Investigation Project and the Communications Project). On balance, the Workfile was considered to be undervalued (except in one school) while the Language Investigation and Communication Projects were considered to be overvalued. The results, however, showed a degree of variation across schools, rendering the results more meaningful at the school level than across the board.

 

Summary

The ESD was developed to provide a standards-based Year 12 programme in English which was independent of unit standards. The schools that opted into the programme identified with the need for a standards-based approach but rejected strongly the unit standards version on the grounds of philosophy, pedagogy and manageability. The schools also wanted a programme which would not require significant adaptation from what they currently offered. The ESD programme was seen to satisfy both of these broad concerns.

The questionnaire to teachers focused on (i) scope and coherence, (ii) workload and manageability, (iii) flexibility, (iv) assessment and (v) moderation. All aspects of the programme were rated favourably by the teachers with satisfaction ratings ranging from 100% (scope, coherence, manageability for staff, administrative manageability and validity of assessment) to 69% (assessment weightings and value of current moderation procedures). Looking to the future, the manageability ratings are particularly relevant given that this is an issue which has been highlighted in the Ministry's Achievement 2001 policy release. The largest dissatisfaction rating was 12% for assessment weightings, although the comments provided by a few teachers indicated that the wording of the assessment criteria requires simplification (this has already been actioned for the 1999 programme).

The questionnaire to students focused on (i) general features of the teaching of the ESD (organisation, quality of course information, stimulation and workload), (ii) development of basic skills (writing, reading, listening and viewing), (iii) assessment (number of tasks, coverage of skills and knowledge, communication of criteria, and feedback), and (iv) value of the individual components of the ESD. By and large students rated the programme less favourably than teachers, however, with the exception of items focusing on the Workfile (encouragement of organisational skills and value for monitoring own progress), satisfaction ratings for items exceeded dissatisfaction ratings, usually by a factor of two or three to one. It is clear from the ratings from individual schools that satisfaction with the programme varied considerably across schools. This suggests a need for schools to interact more regularly in order to share good practice in teaching and assessment.

Reference

Hall, C. (1998) Interim Evaluation of the Year 12 English Study Design Trial 1998. Victoria University of Wellington, School of Education.

 

 

 

 

Sport & Leisure Studies | Graduate Studies | Staff home | Student home | Research | Events calendar

 



(c)opyright 2000 University of Waikato | about