Synthesizing the DLIFLC Conference on High-Level ILR Rating of Speaking

OPI Testing at ILR Levels 3, 4 & 5

ILR Plenary Presentation

September 19, 2008

Monika Ihlenfeld, DLIFLC

Pardee Lowe, Jr., DOD

Elvira Swender, ACTFL

Background

• Increase in number of languages for which

OPI testing is needed

• ACTFL/DLI testing collaboration

– ACTFL trains OPI testers in DLI testing

protocol and ILR rating system

– ACTFL/DLI circulate samples between the two

testing programs

– Maintain rating standardization

New demands for OPI testing

at the higher levels

• Increase in rating discrepancies at 3+

and above

• Using the same ILR Skill Descriptors

– Interpreting them in different ways

• Disagreement regarding what

constitutes an ILR 4

December 2007 Meeting

• ACTFL/DLI Collaboration to address

testing at 3+ and above

• Participants

– DLI OPI Trainers and Master Testers

– ACTFL ILR trained OPI Trainers and Master

Testers

• Arabic, English, French, Mandarin, Persian, Spanish

– Outcome

• Plan a Summit to address the issues of OPI testing

and rating at 3+ and above

• Include government agencies and private testing

contractors

June 2008 Summit

• June 12-13 in Monterey, CA

• 64 participants

– ACTFL, American Councils, Avant Assessment,

DLI-FLC, Department of Defense, FBI, FSI,

Global Linguist Solutions, Language Testing

International, LLE-Language Services,

Monterey Institute of International Studies,

National Foreign Language Center, National

Geo-Spacial Intelligence Agency, National

Language Service Corps, Second Language

Testing, Inc., Washington Language Center

Goals and Procedures

• Observe OPI interview samples at

– ILR Level 5

– ILR Level 4

– ILR Level 3

• Analyze ILR Skill Descriptors of

– ILR Level 5

– ILR Level 4

– ILR Level 3

• Identify the most “salient” features of

– ILR Level 5

– ILR Level 4

– ILR Level 3

*ILR Plus Levels were reserved for a later conference

Day 1 Agenda

• Opening Remarks

– Setting the stage for the discussion

• Focus on L5 (General Session)

– Observe Native speaker performance

– Observe Non-native speaker performance

– General debriefing

• Focus on L4

– Break-out groups

• Observe Native speaker performance

• Observe Non-native speaker performance

– General debriefing

• Groups report back on L4 discussions

Day 2 Agenda

• Focus on L3

– Break out sessions

• Observe Native speaker performance

• Observe Non-native speaker performance

– General debriefing

• Groups report back on L3 discussions

• Focus on Salient Features of L5, L4, L3

– Break out sessions

• Identify and compare salient features of the three levels

– General reporting of groups‟ findings

• Summit Summary and Closing Remarks

What We Learned

Synthesizing the

Features at ILR 3,4,5

Pardee Lowe, Jr.

Charting the Agreement

• Synthesize the results of the four

break-out groups for each ILR Level

• Produce Tables that show

– the features cited

– how many groups agreed on each feature

Difficulties in Synthesizing

• Synthesizing the salient features was

rendered difficult because some

groups cited:

– A whole sentence from the definition;

– Others cited phrases, but not the

complete sentence;

– Still others cited only key words.

Isolating vs. Chunking

the Features

• To determine the amount of agreement,

Lowe isolated the features in each

definition.

• This was admittedly arbitrary, but he cut

them as fine as possible.

• Example: At Level 4 “fluently and

accurately” counted as two separate

features.

ILR LEVEL 5

ILR 5 Speaking

Speaking proficiency is functionally equivalent to

that of a highly articulate well-educated native

speaker and reflects the cultural standards of the

country where the language is natively spoken. The

individual uses the language with complete flexibility

and intuition, so that speech on all levels is fully

accepted by well-educated native speakers in all of

its features, including breadth of vocabulary and

idiom, colloquialisms, and pertinent cultural

references. Pronunciation is typically consistent with

that of well-educated native speakers of a non-

stigmatized dialect. (Has been coded S-5 in some

nonautomated applications.)

Level 5 Definition

• The ILR Level description consists

of:

– 3 sentences, containing

– 12 features

– Unlike Levels 4 and 3, there is no

EXAMPLES section

TABLE 3: ILR LEVEL 3 Salient Features Aligned

* marks items not in the wording of the definitions

Four Groups Agree

Grp 1, 2, 3 & 4 Discourse: All agreed, but characterized it differently, social, professional, extended effective!

Three Groups Agree

Grp 2, 3 &4 Practical, social, professional topics

Grp 1, 2 & 4 Cohesive discourse --If Grp 3's comment is interpreted as "cohesive" -- Four groups may agree

Grp 3 Extended effective discourse

Grp 4 Errors don't interfere, no systematic errors -- If the comments from Grps 1 & 2 are included, 4 agree

Grp 2 Virtually no errors occur in basic structures

Grp 1 Errors virtually never interfere with understanding & errors rarely disturb the native speaker

Grp 2 & 4 Can effectively combine structure and vocabulary to convey menaing accurately -- 3 if Grp 1 include

Grp 1 Use of structural devices is flexible and elaborate

Possible Three Groups Agreeing

Grp 1 & 2 Elaborate concepts freely -- Grp 2 adds: and practically -- If one includes Grp 1, 3 Groups agree

Grp 3 Ability to deal with concepts, to move from personal/concrete to societal/abstract.

Grp 1 Elaborate concepts freely on social and professional topics

Grp 2 & 4 * Supported Opinion -- Perhaps agreement by three if Grp 1's comment is included:

Grp 1 Stating and defending a policy & justifying decisions & clarifying points & answering objections

Two Groups Agree

Grp 1 & 4 Formal/informal conversation

Grp 1 & 4 Normal rate of speech

TABLE 3: ILR LEVEL 3 Salient Features Aligned (cont.)

Cited by Only One Group

Grp 1 Although may not be fully understood can easily repair a conversation

Grp 4 * Abstract discussion

Grp 1 * Hypothesis

Grp 1 Deliver briefings EXAMPLES???

Grp 1 Doesn't miss a beat

Grp 1 Eliciting information from native speakers EXAMPLES???

Grp 1 Incomplete understanding of cultural references

Grp 1 Makes ideas easily understandbale to native speakers

Grp 1 Pronunciaton may be foreign

Grp 3 Recognition and limited use of sociolinguistic features and concepts

Grp 1 Speaks readily and fills pauses suitably without searching for words/phrases

Grp 4 Standard dialect

Participative Listening

Grp 1 Comprehension is quite complete

ILR Level 5 – Agreement

• Consensus that L5 speakers exist

• The features of all four groups taken

together cited every feature in ILR Level

• Three of the four Break-out Groups cited

the “well-educated native speaker”

– One attendee questioned the “concept of the

„well-educated native speaker.‟”

• One group stated “the highest levels of

perfection that the language is capable of”

L5 Sample

ILR LEVEL 4

ILR 4 Speaking

• Able to use the language fluently and accurately on all levels normally pertinent to

professional needs. The individual's language usage and ability to function are fully

successful. Organizes discourse well, using appropriate rhetorical speech devices,

native cultural references, and understanding. Language ability only rarely hinders

him/her in performing any task requiring language; yet, the individual would seldom

be perceived as a native. Speaks effortlessly and smoothly and is able to use the

language with a high degree of effectiveness, reliability, and precision for

all representational purposes within the range of personal and professional

experience and scope of responsibilities. Can serve as an informal interpreter in a

range of unpredictable circumstances. Can perform extensive, sophisticated

language tasks, encompassing most matters of interest to well-educated native

speakers, including tasks which do not bear directly on a professional specialty.

Examples: Can discuss in detail concepts which are fundamentally different

from those of the target culture and make those concepts clear and accessible to

the native speaker. Similarly, the individual can understand the details and

ramifications of concepts that are culturally or conceptually different from his/her

own. Can set the tone of interpersonal official, semi-official, and non-professional

verbal exchanges with a representative range of native speakers (in a range of

varied audiences, purposes, tasks, and settings). Can play an effective role among

native speakers in such contexts as conferences, lectures, and debates on matters

of disagreement. Can advocate a position at length, both formally and in chance

encounters, using sophisticated verbal strategies. Understands and reliably

produces shifts of both subject matter and tone. Can understand native speakers of

the standard and other major dialects in essentially any face-to-face interaction.

(Has been coded S-4 in some nonautomated applications.) Skills Descriptor

Level 4 Definition

• The ILR Level 4 description consists

of:

– 8 sentences, containing

– 30 features, as well as a set of

• EXAMPLES consisting of

– 7 sentences, with

– 33 features

TABLE 2: ILR LEVEL 4 SALIENT FEATURES ALIGNED

* marks a feature not in the wording of the definitions

Four Groups Agreed

None cited

Three Groups Agreed that

Grp 1, 2 & 4 Cultural -- Grp 1 cited "native cultural references"

Grp 1, 3 & 4 Sophisticated

Grp 2, 4 Tailor language/shift to different register -- Three agree, if Grp 1 is included

Where groups have used a slightly different wording I included it in the following:

Grp 1 Sets and shifts tone and register to fit the audience

Two Groups Agreed that

Grp 1 & 4 Effective

Grp 1 & 4 Extensive

Grp 1 & 4 Precise

Grp 1 & 4 Reliable

Grp 1 & 4 Representational/*persuasive --Can advocate a position at length (i.e. persuade and convince) Grp 1

Grp 1 & 4 Smooth -- Fluently Grp 1

Grp 2 & 4 Well-organized discourse, -- Three groups if one includes: Planned and unplanned discourse (Grp 1)

Grp 1 & 2 Rhetorical devices

TABLE 2: ILR LEVEL 4 SALIENT FEATURES ALIGNED (cont)

Cited by Only One Group that

Grp 1 Can perform extensive sophisticated langauge tasks with precision, effectivenss and reliablity…

Grp 1 (cont.) which do not bear directly on a professional specialty

Grp 2 Can tailor language effectively and shift register appropriately in order to accomplish influencing tasks.

Grp 4 Comprehensive

Grp 3 Extensive vocabulary and sophisticated structure allows performance of any task at the appropriate level

Grp 4 Multiple paragraphs

Grp 4 Native

Grp 4 "no" errors

Grp 2 Speaks effortlessly and smoothly and is able to use langauge for the most part with a high degree of effectiveness,

Grp 2 (cont.) reliablity and precision for all representational purposes

Grp 1 Uses appropriate rhetorical speech devices and native cultural references

Grp 1 Uses language fluently, accurately and with precision

Grp 2 Using well-organized discourse can perform tasks that do not bear directly on a professional specialty

Grp 2 (cont.) and can advocate a position at length using sophistircated and rhetroical devices and cultrual references.

ILR Level 4 –

Agreement

The comments of all four groups taken

together: they cited

-- 15 of the 30 features in the

definition, and

-- only 1 of the features from the

EXAMPLES

Agreements at Level 4

Three groups agreed that Level 4 was the level at

which

• The language is sophisticated

• Culture plays a role;

• * Tailoring language is required;

* NOTE: “Tailoring” does not occur in the Level 4

definition. It may be implied by other words at

that level. Compare also: ILR 4 Listening

“understands words specifically tailored…”

ILR LEVEL 3

ILR3 Speaking

Able to speak the language with sufficient structural accuracy and vocabulary to

participate effectively in most formal and informal conversations on practical,

social, and professional topics. Nevertheless, the individual's limitations generally

restrict the professional contexts of language use to matters of shared

knowledge and/or international convention. Discourse is cohesive. The individual

uses the language acceptably, but with some noticeable imperfections; yet, errors

virtually never interfere with understanding and rarely disturb the native

speaker. The individual can effectively combine structure and vocabulary to

convey his/her meaning accurately. The individual speaks readily and fills pauses

suitably. In face-to-face conversation with natives speaking the standard dialect

at a normal rate of speech, comprehension is quite complete. Although cultural

references, proverbs, and the implications of nuances and idiom may not be

fully understood, the individual can easily repair the conversation. Pronunciation

may be obviously foreign. Individual sounds are accurate; but stress, intonation,

and pitch control may be faulty.

Examples: Can typically discuss particular interests and special fields of

competence with reasonable ease. Can use the language as part of normal

professional duties such as answering objections, clarifying points, justifying

decisions, understanding the essence of challenges, stating and defending policy,

conducting meetings, delivering briefings, or other extended and elaborate

informative monologues. Can reliably elicit information and informed opinion from

native speakers. Structural inaccuracy is rarely the major cause of

misunderstanding. Use of structural devices is flexible and elaborate. Without

searching for words or phrases, the individual uses the language clearly and

relatively naturally to elaborate concepts freely and make ideas easily

understandable to native speakers. Errors occur in low-frequency and highly

complex structures. (Has been coded S-3 in some nonautomated applications.)

The Level 3 Definition

• The Level 3 definitions consist of

– 10 sentences, citing

– 38 features, as well as an

• EXAMPLES section, consisting of

– 7 sentences, with

– 37 features

TABLE 3: ILR LEVEL 3 Salient Features Aligned

* marks items not in the wording of the definitions

Four Groups Agree

Grp 1, 2, 3 & 4 Discourse: All agreed, but characterized it differently, social, professional, extended effective!

Three Groups Agree

Grp 2, 3 &4

Practical, social, professional topics

Grp 1, 2 & 4 Cohesive discourse --If Grp 3's comment is interpreted as "cohesive" -- Four groups may agree

Grp 3 Extended effective discourse

Grp 4 Errors don't interfere, no systematic errors -- If the comments from Grps 1 & 2 are included, 4 agree

Grp 2 Virtually no errors occur in basic structures

Grp 1 Errors virtually never interfere with understanding & errors rarely disturb the native speaker

Grp 2 & 4 Can effectively combine structure and vocabulary to convey menaing accurately -- 3 if Grp 1 include

Grp 1

Use of structural devices is flexible and elaborate

Possible Three Groups Agreeing

Grp 1 & 2 Elaborate concepts freely -- Grp 2 adds: and practically -- If one includes Grp 1, 3 Groups agree

Grp 3 Ability to deal with concepts, to move from personal/concrete to societal/abstract.

Grp 1 Elaborate concepts freely on social and professional topics

Grp 2 & 4 * Supported Opinion -- Perhaps agreement by three if Grp 1's comment is included:

Grp 1 Stating and defending a policy & justifying decisions & clarifying points & answering objections

Two Groups Agree

Grp 1 & 4 Formal/informal conversation

The Level 3 Agreement?

• All four groups taken together cited

a total of 13 out of 38 features in

the definition;

• Only 3 out of 37 features in the

EXAMPLES

ILR Level 3 –

Agreement

Four groups agreed that Level 3 was the level of discourse;

Three groups agreed that discourse was cohesive; the fourth

group had a wording that could have been so interpreted;

Three groups also had comments about errors, although their

wording varied:

Errors don‟t (virtually never) interfere with

understanding (and rarely disturb the native speaker);

Virtually no errors occur in basic structures;

No systematic errors

Three groups cited: Can effectively combine structure and

vocabulary to convey meaning accurately; if one includes:

Use of structural devices is flexible and elaborate

ILR Level 3 –

Agreement– Cont-1.

AT THIS POINT WE CITE THE STATEMENTS OF GROUPS TO

SHOW HOW COMPLICATED ASCERTAINING THE SALIENT

FEATURES CAN BE:

POSSIBLE AGREEMENT OF THREE GROUPS ON “CONCEPTS”:

Elaborate concepts freely – two groups;

and practically – adds on of those groups;

Ability to deal with concepts, to move from

personal/concrete to societal/abstract – a third group;

Elaborate concepts freely on social and professional topics

– added by an above group;

ILR Level 3 –

Agreement– Cont-2.

ON * “SUPPORTED OPINION”: Two groups mention this feature, three

if another group’s comment is included:

Stating and defending a policy & justifying decision & clarifying

points & answering questions

* NOTE: There is no wording on Supported Opinion in the ILR Level 3

Speaking Skill Description itself,

BUT it might be implied by the EXAMPLES that follow there and from

the actual wording in the ILR Reading and Listening Skill Level

Descriptions – both of which cite understanding Supported Opinion.

FEATURES MENTIONED

BUT NOT IN LSLDs

FEATURES CITED BUT NOT

IN LSLDs @ ILR 4

• Level 4: Comprehensive;

• Level 4: Multiple Paragraphs;

• Level 4: Native;

• Level 4: No Errors;

• Level 4: “Can tailor language effectively

and shift register in order to accomplish

influencing tasks”;

FEATURES CITED BUT NOT

IN LSLDs @ ILR 3

• Level 3: Supported Opinion;

• Level 3: Hypothesis;

• Level 3: Abstract Discussion;

• Level 3: Abstract Linguistic Formulations;

• Level 3: “Recognition and limited use of

sociolinguistic features and concepts.”

FURTHER STUDY

Aspects Warranting

further Study

• Nature of the “highly articulate, well-

educated native speaker”;

• Difference between Level 4 Representation

and Level 3 Supported Opinion;

• Features groups cited that were not

directly mentioned in the ILR LSLDs;

• Aspects of research since the ILR LSLDs

were revised in 1985?

Next Steps

Conferences

• Two per year

• Future Topics

– Definition of Terms

• cf. ACTFL Speaking‟s Explanatory Appendix

– Plus Levels

– Other Skill Modalities

• Reading

• Listening

• Writing

Bigger Picture

• Implications for the ACTFL

Guidelines Revision 2010

• Collaborating on any future revision

of the ILR Skill Descriptors for

Speaking

The Future of OPI Testing at

the Higher Levels?