Measuring Progress in Discourse Production. A genre-based approach

The aim of this paper is to find criteria for the evaluation of oral business discourse produced in the language laboratory by students of business German. I felt the lack of an instrument that measures more precisely, not only linguistic criteria like lexis and grammar, but also the pragmatic background and the functions that discourse types within ‘language for specific purposes’ (LSP) have to fulfil. By evaluation I mean the asses-sment of individual progress, but indirectly I evaluate a teaching concept. A good teaching programme should facilitate progress. I am going to describe a teaching concept called Ubertragung ‘text transfer’, focusing on the project that provided the data material for the present study. I will then present the model I have chosen as a basis for the study. In the study itself I concentrate on ‘text organisation’ and ‘content structure’.


Teaching concept and project description
The teaching concept Übertragung 'text transfer' was designed for the training of oral skills in business communication with adult learners. It involves reading an input text, for example a newspaper article in the native language. For our project we chose a newspaper interview with a successful Norwegian furniture producer. This text provides the content for the exercise. Students are also given a task description, i.e. instructions for a business situation they are to simulate as speakers. The description contains information on the situational context expected in pragmatic linguistics (1) role and status of speaker and addressee (2) location (3) formality level (4) medium and channel (5) subject matter (cf. Levinson 1983: 23). The students are instructed to take the role of the furniture producer who has been asked to ring up a German im-porter of quality furniture and to give him details about the product range (referred to as product), the delivery potential (delivery) and the economic situation in Norway and Germany and how that influences the producer's competitive status (economic situation). The speaker is, of course, expected to realise that he is talking to a potential customer. He has to perform a professional task implying strategic language use. The result or output text is recorded on tape in the language laboratory.
The corpus for the project consists of 20 transcribed output texts from a test at the start of the course (test 1) and 45 texts from the final exam (test 2). During the four months between tests students attended 10 two-hour training sessions. A description of the teaching process lies outside the scope of this paper.
When speakers produce output texts, they go through a cognitive process, the mental processing of content. The input text is decoded and stored in the mind, then the mind is active planning the output and finally the output text is encoded and spoken. Prior knowledge puts constraints on the way every individual stores knowledge and later reproduces it. This abstraction has its background in schema theory (cf. Kintsch and van Dijk 1978).
Somewhat simplified the activity I have described is sending on a message. The process has parallels with translation. Written text is transformed into spoken discourse, native language into foreign language, written monologue into simulated interaction, business content into business communication, text reception into text production. This parallel with translation makes many learners focus on translation strategies, translating large passages of the input text and forgetting the communicative situation. I call this an atomistic approach. It makes speakers lose the overview and the core of the information. They focus on the linguistic details rather than on the business strategy. It is therefore essential for the teacher or trainer to inspire students to set up a plan or strategy for the message and only then go into the details of formulating concepts and applying grammatical rules. This I call a holistic approach.
When trying a holistic approach to evaluation we run into difficulties. It is easier to quantify and assess linguistic criteria on the sentence level like lexis and grammar. I also tried quantitative machine evaluation and found for example that the average student increases her word per minute ratio by 24,53 per cent between the two tests. This says very little about the treatment of content, which is one of my aims. I think it is time to go away from a predominately atomistic or quantitative approach to a top-down holistic approach, subordinating linguistic criteria under criteria of a holistic perspective. I therefore propose a genre-based approach.

The linguistic base
In recent pragmatic linguistics there is general agreement that text genres are to be classified according to their communicative purpose. Swales' (1990) and Bhatia's (1993) approach to the production and analysis of genres is very practicable from my point of view. Let us look at Bathia's definition of genre, based on Swales' works. Genre is: "A recognisable communicative event, characterised by a set of communicative purposes, identified and mutually understood by the members of the professional or academic community in which it regularly occurs.
Most often it is highly structured and conventionalised with constraints on allowable contributions in terms of their intent, positioning, form and functional value.
These constraints, however, are often exploited by the expert members of the discourse community to achieve private intentions within the framework of socially recognised purpose(s). (Bhatia 1993: 13) The term communicative purpose is not unproblematic. Although the members of a professional community would know what to say and how to say it, they may not be aware of the full set of communicative purposes. Swales refers to examples from law procedure and political speeches. Cross-examination of witnesses is more than just eliciting the facts of the case. By skilful question techniques lawyers may control what a witness reveals and what not in favour of the one side. Politicians in their speeches not only present party policies, but they will also ridicule opposition parties and generate applause (cf. Swales 1990: 46-47). The researcher as an outsider has the difficult task to investigate these complex purposes. For the present study I will focus on the situation according to the task description and general professional business goals. These goals have earlier been presented to the students as a norm. During text production there may be a conflict between the communi-cative purpose of the simulated business situation and the 'actual' testclassroom situation. This I call the 'simulation effect'. It must therefore not be neglected. Before I expand on the business goals and the simulation effect, we have to look at the steps in Bhatia's model. For Bhatia the communicative purpose is the main discriminating factor in a genre. The general orientation for discourse production and discourse analysis is linguistic, psychological and socio-cultural, implying that in the mental process of discourse production these three dimensions are activated. Consequently we pay attention to these three dimensions in the analysis of discourse. I interpret this as displayed in figure 1.

Figure 1: Proposed model for text production and analysis
A step in Bhatia's analysis is the structural interpretation of a genre. In his examples text structure can materialise as a linear organisation or sequence of moves or actions. In a sales promotion letter he thus identifies the moves: (1) establishing credentials, (2) introducing the offer, (3) offering incentives (4), enclosing documents and (5) soliciting response (6) using pressure tactics (7) ending politely (1993: 46f.). This text organisation is a text convention, a socio-cultural feature that is expected in a "discourse community". Within the constraints of text conventions the author or speaker then employs individual strategies to produce discourse in a way to serve her or his purpose. Individual stra- To my mind Bhatia's approach is dynamic because it acts on the tension between socio-cultural constraint and individual freedom in speech production. From a pedagogical standpoint we evaluate texts with both orientations in mind, not forgetting linguistic realisation. I have done that by looking at text organisation and content structure. Text organisation I see as the linear sequencing of moves or actions, following text conventions. I prefer the term action to move, as Bathia's move concept seems to go further than mere linear organisation, for example in his 'cognitive structuring ' (1993: 115). I have segmented the text into actions heuristically, according to my knowledge of business conventions. Content structure I see as the internal structure of content, here materialised in the cognitive or strategic management of the three topics in the exam task: product, delivery and economic situation. I found more variation in content structure than in text organisation.

Communicative purpose and business goals
I said above that business goals would be focused on in the evaluation. We can distinguish between the current business situation and general business goals. In the current situation the Norwegian furniture producer has been asked to ring up a German importer and to give him some information. General business goals can be expressed in meeting customer needs, i.e. providing goods and services, or in capital gain for the producer. The producer and supplier has to start activities in order to be able to provide those goods and services. The overall communicative purpose is persuasive, influencing the other person to agree to something that gives you a profit, in the actual case it is buying furniture. Written and oral communication is an essential activity in such business transactions. Rules for such communication are laid down in negotiation theory and handbooks on sales talks. The following is a list of essential qualities that are needed in contacts with business partners in order to attain the communicative purpose (cf. Fisher and Ury 1981, Fisher and Ertel 1997, Lewicki and Litterer 1985, 1994, Wage 1991). Bhatia similarly sets up 'such functions' for sales promotion letters (1993: 45-46). The following list is part of the teaching programme: -Planning, strategy. You have to plan your talks carefully, in line with the business strategy.
-Preparation. You have to be well prepared, using relevant information and using it skilfully.
-Clarity, efficiency. You have to be clear and efficient, saving time and money.
-Interaction. You need to have good interactive strategies: giving feedback, explaining, arguing and asking questions -Cooperation. You have to be cooperative, building good relationships with your business partners, and at no costs must you act in a way to risk losing a good business relationship.
Linguistic theory has parallel rules for general communication. Business communication goals harmonise with Grice's cooperative principle (cf. Levinson 1983: 101). It seems that for business communication the focus is to a higher degree on the results of a process than on the process itself. Turning to speech act theory, the communicative purpose has a parallel in the main text illocution.

Communicative purpose and simulation
There is always the danger that simulations disturb the communicative purposes and that the output texts show signs of the simulation effect. I have tried to demonstrate that in table 1. In the left column, the title 'situational background' covers the communicative purpose and pragmatic context criteria for context (cf. Levinson's criteria, 2 above). With these context criteria students are supposed to produce a 'real life business situation' (middle column), but in actual fact they simulate in the language laboratory (right column) performing an exercise or a test that is assessed by the teacher. I call the effect of this process simulation effect. Even though Übertragung is generally evaluated highly as a training concept, the simulation effect cannot be neglected in an evaluation. In the present paper I have to leave it at this reference.
There is, on the other hand, one great advantage in simulations that may help us to learn something about text production. We know the task and the input content and we can study and evaluate how it is exploited for the communicative purpose.

Text organisation
It is not my aim to define a genre in this paper. Nevertheless the speakers have performed a task, consisting of a 'set of communicative purposes', and they have simulated a 'communicative event'. These are features in Swales' and Bhatia's definition of genre (cf. above). It allows us to apply genre analysis. For that purpose I have segmented the text heuristically into actions (cf. footnote 4). I group these actions as is customary in conversational analysis in an opening phase, a main phase and a closing phase. The transition from the opening phase to the main phase and the signal anticipating the closing phase are crucial. In negotiation theory and salesmen's manuals we recognise these phases as a potential for tactical behaviour. Initiative at the transition to the main phase can give the speaker a strategic advantage and the signal anticipating an important decision or negotiation agreement marks the initiative to start the closing phase.

Observations and tendencies
In the first test students were not aware of phases and actions. Many speakers forgot an opening phase and a closing phase. They just transferred the message, forgetting the addressee. Text organisation is relatively easy to teach, train and assess and we can clearly notice that the speakers have a better command of text conventions in the second test, i.e. they have made progress. There is limited individual variation in the number and relevance of the actions and in the order of the different actions.

Transition to main phase
Linking topic in phase 1 with main phase. In the example (8)

Main phase
Introducing following information/message. (Example 10)

Structuring following information. (Example 11)
Main body of message. The focus is on three topics, cf. the section on content structure below.
(10) und eh ja ich hab jetzt hier die Auskünfte um die Sie gebeten haben 015 ' and here I have the information you asked for' (11) also erstens werde ich Sie unsere Produkte nennen und dann werde ich über die Liefermöglichkeiten an Ihre Firma sprechen drittens ... 023 'first I am going to talk about our products and then I will talk about delivery conditions'

Signalling the closing phase
Addressee's name followed by short pause. This is normally an introduction to one of the following actions. 005 (Example 12).

Observations and tendencies
We saw above that there does not seem to be great individual variation in text organisation. That may be due to students' following a normative pattern or, according to Bhatia's definition of genre, there are high "constraints on allowable contributions" (1993: 13). Content structure, on the other hand, shows considerable variation. Speakers are free to find the relevant topics in the input text and present them in a way to satisfy the communicative purpose. Therefore greater variation in content structure does not surprise. This harmonises with my assumption that there is more scope for individual strategies in content structure than in text organisation ( fig.1). Another reason may be that different prior knowledge affects the cognitive process in different ways. We can observe a tendency in learners to progress from an atomistic to a holistic approach in the course of study. This means that there is more evidence in test 1 of people concentrating on the details in the input text, rather than concentrating on the communicative situation. However, progress in topic management is not as striking as in text organisation. I can see several reasons for that. First it is difficult to compare two different tests with different content. Secondly, it is not as easy to teach content structure as text organisation. Speakers with little training and poor motivation perform less well in both tests. A third reason is that it is easier to teach socio-cultural skills than it is to influence the mental operations that are necessary for good topic management.

Criteria for evaluation
Below I suggest a list of 10 evaluation criteria. Reference to relevant business rules as stipulated above, is given in each case. On the macro level I have studied the interaction between the topics production, delivery and economic situation. On the micro level I have studied their inner structure. Poor performance, more often found in test 1, is opposed to acceptable performance, more often found in test 2. Progress is to be seen along a scale between the two.

Macro level criteria
Topic order. In test 1 I noted quite a few instances where students followed the topic order of the input text, disregarding the actual situation. In test 2 students to a greater degree followed the actual situation, i.e. the order of questions put by the German importer. This is particularly important in topic 3, where the task was to connect two topics of the input text, competitiveness and the economic situation. From the business perspective this displays the quality of planning.
Cohesion. In test 1 topics were often listed like beads on a string, without connections to the actual situation. In test 2 there were more instances where topics were linked with each other, underlining their relevance in the situation or following some argumentative intention. This can be interpreted as an indication of a sound business strategy. Split topics. In test 1 there were more examples of topics recurring without reason, indicating poor strategy and preparation. This may have been rational in the input text, but irrelevant in the current business situation. In the next example the speaker starts like this: (24)  Irrelevant topics. In test 1 more topics from the input text were irrelevant for the present business situation than in test 2. In a business talk this can be interpreted as bad preparation.
The introductory reference to a foregoing request in example (29) displays good preparation as well as cooperation: (29) und eh ja ich hab jetzt hier die Auskünfte um die Sie gebeten haben ja die Produkte die wir für den Export herstellen ... das sind vor allem Sessel ... 015 'and here I have the information you asked for yes the products we produce for export are mostly chairs' Correct information. Students at this stage do not seem to have great difficulties understanding written text. Difficulties may arise when economic content is not understood. Exchange rates caused some confusion and brought about deficient information. As preparation is essential in business communication, such 'non-information' has to be assessed negatively.
Linguistic realisation. The progress made in lexis and grammar is notable between the tests, but in the current context this is evaluated in terms of the realisation of genre criteria.
Topic layers. The most interesting criterion is what I propose to call 'topic layers'. A topic can sometimes be treated very superficially, one layer only, in that it is just mentioned, but not expanded on. This certainly does not serve the intended communicative purpose. Thus one speaker says about delivery: (30) ich finde daß die Möglichkeit an Sie zu liefern sehr gut ist 002 'I think that the possibility to deliver to you is very good' and no more. More advanced 'multi-layered' utterings carry more information. The argument can be developed in many different ways. Topics are for example introduced, refer to the partner's request for information, argue in favour of his needs or in favour of a quality of the product, and connect this with an advantage for the partner. This can be seen in example (31) 'you would like to know what products we can offer? (introduction of topic) you know our Stressless don't you (introducing product) that is a chair (specifying) and it has become a success (additional information) and we have also protected its name (additional information) furthermore we also sell beds, sofas and sofa groups' (answers partner's request) The speaker tries to satisfy his partner's request for information and makes the most of the little information there is in the input text.
Self-related or other-related. How does a speaker observe the rule of cooperation, in order to build up a good relationship with a customer? The one-layered example (30) is self-related. The speaker says that he himself considers the possibility to deliver to be good. This may not be reason enough for a potential customer to buy the goods. In the multilayered example (32) the producer refers to the customer's mind or interest (as you may know; more expensive for you), trusting him to be well informed. He also explains the economic situation step by step. This is 'other-related', showing concern for the customer's interests and building up a cooperative relationship.
(32) wie Sie vielleicht schon wissen sehen die Konjunkturen in Norwegen und Deutschland anders aus die norwegische Krone ist ja aufgewertet worden also die norwegische Krone ist mehr wert gegenüber vorigem Jahr und das bedeutet daß unsere Möbel teurer geworden sind für Sie Herr Münzig 'as you may know the economic situations in Norway and Germany are different the crown has been revalued as you know that is it is worth more than last year and that means that our furniture has become more expensive for you Mr. M.'

Conclusion
This study has shown that genre analysis can successfully be applied in the evaluation of simulated business discourse. Building on Swales and Bhatia I studied the two features text organisation and content structure, with special focus on criteria enhancing business relationship. Learners show a tendency of less individual variation and more progress in text organisation and more variation and less progress in content structure. That strengthens the claim laid down in my evaluation model that text organisation depends to a higher degree on socio-cultural conventions (figure 1). Such conventions seem to be easier to learn. Content structure, on the other side, shows much more individual variation. Although content has to be presented in a manner respecting genre conventions -in our case it is the business message and general business rules -the individual speaker seems to have much more freedom to structure her content. This strengthens the assumption that psycholinguistic aspects play a more important role in content structure.
As to the evaluation of the teaching concept programme itself, the lower degree of progression in content structure implies that learners need more help on how to handle available information and on the implications for the business relationship. They must increase their awareness of the business situation, what topics to choose and how to arrange and present them to the listener.
Two different tests were given to the same population, before and after the course. This proved disadvantageous for the study of content structure. The same test given to two different populations, before and after the course respectively, may provide material for a more accurate comparison.
The simulation effect was not studied systematically in this project. It should be given priority when more comparable data are at hand.