The TQA Tool, a four-page rating sheet, is a unique and ground-breaking approach to evaluation of translations. The tool, grounded in translation theory, builds on the experience of existing translation quality assessment approaches and is designed to overcome known limitations (e.g., arbitrary point systems, narrow evaluation foci).

The tool draws on functionalist theory and textual and theoretical models that focus on the reader’s response, practical aspects and linguistic features of the translated text, and the relationship of the translation to the source text. Functionalist approaches recognize the role of extra-linguistic features (those not included within the realm of language) and promote the idea that the form (text type) and function or purpose and use of a target language text is of paramount importance in determining translation quality. Evaluations conducted by professionals with advanced language skills, trained to use the TQA Tool take into consideration various aspects emphasized in other current approaches to translation quality.

A Compositional Tool

The TQA Tool uses four categories of assessment, with each component emphasizing a different aspect of translation quality. In this regard, the TQA Tool is a compositional tool designed to evaluate the four key aspects of any translation. These components include:

• Target Language – an examination of the quality of the translation’s linguistic form (e.g., spelling, grammar, lexicon)
• Textual and Functional Adequacy – a consideration of how well the translation achieves the goals, purpose and function of the text for its target audience
• Non-Specialized Content – an assessment of whether the content accurately reflects the original text.  
• Specialized Content – an appraisal of translation accuracy of special terminology and/or specialized content.

The aim of the tool is to give requesters and health care decision makers a general overview of the quality of a translation product (i.e., the assessment purpose is summative).  Requesters receive from TQA raters a summary evaluation of the translation’s overall quality in the form of a recommendation about the adequacy of the translation.

The object of assessment is the product, or the translated text. The goal is not to determine why the translator made a certain mistake, but rather what effects the mistake has on the text and its intended meaning. Although the TQA Tool has been tested specifically with health care education materials, it can also be used to evaluate other translation types.

A Textual Approach

The TQA Tool uses a textual approach to evaluation, meaning that texts are seen as whole units, comprised of smaller linguistic sub-units (words, headings, sentences, paragraphs). The basic premise of an evaluation is that texts are organized in a particular way to achieve a defined function (to educate, inform, explain) within a particular purpose for an intended audience (communicative context). In this regard, texts are the result of the writer's assumptions about the audience, as well as the audience's contributions within a particular communicative context (usually in the form of previous knowledge, attitudes or beliefs). The translator must work to recreate the text for a new target audience, which is often different from the audience envisioned by the author of the original text, and for a new communicative context which is sometimes in conflict with that of the original (Colina 2003).

Simply put, translated texts are not independent or objective entities. The same text can be translated in different ways depending on the communicative context of the target text or the assumptions made about the audience. In evaluating the quality of a translated text, raters are instructed to not only look at the linguistic sub-units, but also to how well the text achieves its overall communicative purpose. In other words, does an educational brochure work as such; does an ad sound like an ad and function like one? Finally, translation errors are not considered isolated linguistic or grammatical errors (e.g., agreement, conjugations, use of tense and mood with verbs, lexical choices) but instead should be evaluated based on the overall effect they have on the text and its communicative function.

Descriptive Statements for Classification

Rather than using a system where points are assessed for each error, somewhat arbitrarily, the TQA Tool uses descriptive statements to classify texts into one of four assessment categories. Each descriptor defines the degree of compliance with specific criteria. Raters evaluate text by matching their assessment to a descriptive statement for each evaluation category. The goal is to choose the statement that best describes the text in each of area of evaluation. Descriptors reflect a four-point range, from unacceptable to ideal, for each component evaluated.

• Target Language – The text is: 1) extremely difficult to read (incomprehensible); 2) hard to comprehend, (includes elements/structure from the source text); 3) generally readable but with awkward expressions; or 4) reads similar to texts originally written in the target language.

• Textual and Functional Adequacy - The text: 1) does not consider the goals, purpose, function and audience (cannot be repaired); 2) gives some consideration to the intended purpose and audience (repair will require considerable effort); 3) approximates goals, purpose, function and needs of the intended audience (can be repaired with edits); or 4) accurately accomplishes the goals, purpose, and function of the original (minor or no edits needed).

• Non-Specialized Content – The text: 1) contains important unwarranted deviations from the original (very defective); 2) exhibits some misunderstanding of the original and/or the translation instruction; 3) minor alterations in meaning, additions or omissions; or 4) accurately reflects the content contained in the original.

• Specialized Content – The text: 1) reveals an unawareness/ignorance of special terminology and/or insufficient knowledge of specialized content; 2) contains serious/frequent mistakes involving terminology and/or specialized content; 3) has a few terminological errors, but the specialized content is not seriously affected; or 4) is accurate and appropriate.

During the evaluation process, the translation is examined four times. Each time, a rater looks to match the text to the best description in each of the four categories. However, raters are instructed to first assess the translation for Target Language and Textual and Functional Adequacy without looking at the English original text. This enables raters to capture a first impression, as the target reader is likely to gain from reading the translation without reference to the source text.

Raters are then asked to read the source text, contrasting the translation with the source text and then with their first impressions as a target reader. This step is vital as a measure of how well a translation anticipates and honors the communicative context of intended target audiences. Consequently, raters may need to adjust their first rating before continuing the assessment for the final two categories – Non-Specialized Content and Specialized Content.  With each rating component, raters emphasize a different aspect of translation quality during their assessment.  Considering all four aspects raters are then ready to make a recommendation about the translation.