Questionnaire translation in the European Company Survey : Conditions conducive to the effective implementation of a TRAPD-based approach

The European Foundation for the Improvement of Living and Working Conditions (Eurofound) carries out three recurring Europe-wide surveys. Up until 2013, the quality control of the questionnaire translation for these surveys was performed using back-translation. In the 2013 edition of the European Company Survey (ECS) – an establishment-level telephone survey – Eurofound for the first time followed a modified version of the Translation, Review, Adjudication, Pretesting, and Documentation (TRAPD) approach to questionnaire translation. This paper outlines why the TRAPD approach was adopted and how it was operationalised – resulting in a modified TRAPD implementation – for the purposes of the ECS 2013. It provides a reflection on the conditions that proved particularly conducive or obstructive for its efficient and effective implementation and the lessons that can be drawn for future surveys.


Introduction
The aim of this paper is to outline how and for what reasons a modified version of the Translation, Review, Adjudication, Pretesting, and Documentation (TRAPD) approach to survey questionnaire translation (Harkness, 2003) was adopted by the European Foundation for the Improvement of Living and Working conditions (Eurofound) for the first time for the 2013 edition of its European Company Survey (ECS 2013).The paper is structured as follows.We first discuss the importance of the concept of equivalence in cross-cultural research and discuss some of the solutions that have been proposed to address the issue.We then review back-translationbased as well as collaborative approaches to assess translation quality, before outlining the translation approach Eurofound applied in, and developed across, its various surveys.We next provide a detailed description of the transition from a back-translation-based approach to a modified version of TRAPD, and discuss the practicalities of implementing it for the ECS 2013.We conclude the paper with an assessment of the extent to which the approach was implemented efficiently and effectively, identifying those aspects of the approach and of the implementation process that were particularly conducive or obstructive in this regard, and providing some recommendations for future research.

Cross-cultural research and problems of equivalence
Cross-cultural research in the social domain has increased considerably over the last few decades as a consequence of the globalisation of the economy, the processes of regional integration, and the (related) needs for evidence and comparison between different cultural contexts.The methodological issues specific to cross-cultural research have been acknowledged and discussed extensively by researchers in numerous social research domains, but often theoretical and methodological reviews of cross-cultural management research take a rather pessimistic tone (Sekaran, 1983).
An important reason is the equivalence between research instruments that is required to reliably compare cultures or countries.Regardless of their topic of interest, researchers face the issue of whether their results can be reliably and meaningfully compared or are like 'apples and oranges' (Stegmueller, 2011).Most researchers are fully aware of this and acknowledge the existence of country heterogeneity in attitudes and preferences.Equivalence in crosscultural research is therefore a major concern for numerous researchers, who consider it crucial to ensure reliability and meaningfulness of findings.To compare data from different nations, cultures or groups of populations, and to avoid mistaking methodological artefacts for real differences, researchers must ensure that the data and ultimately the results of analysis are equivalent.That is, any errors should occur in the same way in all spatial units (Baur, 2014), or, depending on the comparisons researchers are looking to make, across cultures or across groups.Applied to measurement error, this implies, for instance, that if a questionnaire item slightly overestimates a phenomenon in one country, and it is not possible to address this error, the item should result in the same overestimation in all countries, to allow for the comparison of the results across those countries.
A key aspect of cross-cultural equivalence is construct equivalencewhich implies that a given concept or behaviour must have (approximately) the same meaning and function in all contexts and cultures under study to allow for comparison (Douglas & Craig, 2007;Hult et al., 2008;Singh, 1995).As Singh (1995) points out, construct equivalence is a notion rooted in the etic (i.e.universal) perspective that encompasses simultaneously two types of equivalence.On the one hand functional and conceptual equivalence, which refer to the extent to which a phenomenon serves the same function and is expressed similarly in different cultural contexts.On the other hand instrument and measurement equivalence, which refer to the extent to which the operationalisation of the phenomenon (that is, scale items and response categories) is interpreted similarly and measures the underlying construct equivalently in cross-cultural data.Hult et al. (2008) argue that in crosscultural research, both etic (i.e.universal) and emic (i.e.culture-specific) measures can be used to equally represent the theoretical domain of a construct across cultures.Rigorous cross-cultural research should therefore be able to capture both etic (i.e. the commonalities) and emic (i.e. the uniqueness) in the meaning of constructs in and across cultures (Hult et al., 2008).
Whereas measurement equivalence can only be established analytically after the data has been collected, instrument development and translation should be designed such that it ensures functional, conceptual, and instrument equivalence.Simply put, in order to interpret and understand variations between characteristics that are universal across countries (i.e.employees working overtime), research needs to capture at least some information on the specific characteristics that shape them (i.e. in some cultures working overtime would be regarded as positive -an expression of commitment -and in other cultures would be regarded as negative -inability to carry out the work within the designated time).Reversely, and applying the distinction to questionnaire construction and translation, if comparisons are to be made, the concept that is compared needs to be truly universal, and any variability in specific and local interpretations of the concept should be minimised (i.e. when comparing reported overtime across cultures, researchers need to be aware of these differences and their likely impact on reporting).A lack of equivalence threatens the validity of substantive inferences in cross-national research (Hult et al., 2008;Renschler & Kleiner, 2013;Sekaran, 1983;Singh, 1995).

Ensuring equivalence in cross-cultural research
Researchers undertake various efforts before, during, and after data collection to ensure equivalence.Survey researchers harmonise the whole "survey cycle" to achieve equivalence and therefore comparability.That is, they try to ensure that every interviewed person is asked the same questions, in the same order, and using the same response categories; they try to use the same respondent recruitment strategy and create the same interview situation as well as use standardised and replicable statistical analysis to arrive at their results (Baur, 2014).
When designing the research and developing the research instruments, the first element to be considered is what concepts researchers want to measure and how to measure them equivalently.The issue is that -even when a theoretical concept is clear to the researcher -it can mean very different things to different respondents in different cultural contexts.This is more likely to be the case when concepts are more abstract (Scheuch, 1993).Therefore, the first steps in developing cross-cultural (survey) research instruments are typically semantic analysis and dimensional analysis (Baur, 2014).
Semantic analysis aims at assessing the meaning of concepts in each of the languages in which the research instrument needs to be made available and ensure that these concepts can be translated appropriately.
Dimensional analysis starts from the observation that very often concepts are latent variables -that is, variables that are not directly observed but are rather inferred from other variables that are observed -and are therefore better measured by using multiple indicators.In the words of Baur (2014), "dimensional analysis asks which aspects are part of the concept and how these indicators can be measured best" (p.268).For each of these indicators, questionnaire items need to be constructed.
When formulating these items, several types of equivalences should be taken into account, such as vocabulary equivalence or a translation that is equivalent to the original language in which the instrument was developed; idiomatic equivalence which could become a serious problem when some idioms unique to one language just cannot be translated properly in other languages; grammatical and syntactical equivalence, which is especially important when translating long passages; experiential equivalence or the equivalence of the inferences drawn by the respondents in various cultures from a given statement […].All the above types of equivalence refer generally to the equivalence of source and target versions of the instrument.(Sekaran, 1983, p. 62) Translation of cross-cultural research instruments therefore plays a crucial role to ensure equivalence and comparability of data collected in different countries or in different lingual and cultural groups.It should be carried out by translators thoroughly familiar with the source and target languages as well as with the cultures involved (Sekaran, 1983).

Questionnaire translation and the challenge of achieving equivalence
A good questionnaire translation should, on the one hand, take into consideration the different social realities, cultural norms, and respondent needs (e.g. level of vocabulary) existing within and across countries and, on the other hand, respect the questionnaire design and retain measurement properties (e.g. using the same or equivalent scales; Harkness et al., 2010).A good questionnaire translation must go beyond the systematic transposition of questionnaire items from the source language into the target language.1Not all of the methods that are used to assess the quality of translations are equally geared to assuring this.
In the social research domain, back-translation has been widely used to assess the quality of translation in cross-cultural research.Back-translation was considered the main technique for the quality control of translations for many years (Scheuch, 1993).In the back-translation procedure, a bilingual translates the research instrument (i.e. a questionnaire) from a source language into another (target) language.A second bilingual then translates the research instruments back into the source language.The original and back-translated versions are then compared to identify differences and to assess comparability.The accuracy of the back-translated version of the research instrument(s) is considered an indicator of the accuracy of the target translation (Brislin, 1970;Douglas & Craig, 2007;Mullen, 1995;Scheuch, 1993).Back-translation was originally developed to deal with situations in which the researcher is not familiar with the target language but requires assurance about the fact that respondents are being asked the same question in that language (Harkness, 2003).Although back-translation can potentially identify translation errors, it is subject to many limitations which reduce its usefulness.For instance, it does little to control one of the key problems in question wording, which is the equivalence of meaning (Douglas & Craig, 2007;Scheuch, 1993).As backtranslation assists in obtaining a direct or literal translation from one language to another, it is possible to move from one language to another and back again without capturing the intended sense of the statement.Direct translation between languages assumes that words have a concrete referent in reality, which allows for exact correspondence between languages.It overlooks the cultural embededness and mediating function of language (Maclean, 2007).Therefore, the real meaning of a statement and consequently the equivalence of meaning is missed as back-translation is not able to catch such cultural difference.Even in catching errors, much depends on the back-translator's understanding of a question and its purpose (Douglas & Craig, 2007).Furthermore, even when back-translation shows differences between the source and the back-translated version, it can sometimes be difficult to tell whether this is due to the original translation, the back-translation, or both (Hanson, 1968).These limitations suggest that back-translation may not be the optimal approach to produce an instrument that can validly and reliably be used across languages.Back-translation is too much oriented at a direct, literal transfer of one language into the other, which is undesirable in its own right, but also renders the process ineffective for detecting issues with differing meaning across languages and cultures, and provides no opportunities to improve the source questionnaire.
To overcome the limitations of back-translation, various procedural solutions have been proposed and alternative approaches developed.For example Bairoux (1948) sought to eliminate the problem of lack of equivalence that can stem from back-translation by designing a more complex procedure that entailed multiple translators and reviewers, as well as an exhaustive set of notes about the objective of each question to ensure that translations were based on meaning rather than wording (Barioux, 1948).Brislin (1976) warned that back-translation can give a false sense of security and suggested that a 'decentring' procedure be adopted.A 'decentred' approach implies that rather than starting from a single culture or language, research instruments should take into account the idiosyncrasies of all the languages under study (Brislin, 1976).With the aim to address issues of equivalence, decentring refers to a translation process in which the original and the second language versions are both subject to modification.That is, the original language version does not contain content that must be translated without change into the second language.This procedure allows for the modification of words and concepts that have no clear equivalents in the other language (Chapman & Carter, 1979, p. 72).
Simultaneously, a number of collaborative or team-based approaches designed to overcome the limitations of back-translation emerged.They aimed to tackle issues to do with the complexity and subjective nature of translation by involving teams in which members would bring in a mix of skills and disciplinary expertise, combining substantive knowledge of the study at hand, knowledge of questionnaire design as well as cultural and linguistic skills to translate the questionnaire (Douglas & Craig, 2007).A team can bring together the mix of required skills and the disciplinary expertise required and the team members "need to have knowledge of the study and of questionnaire design, as well as the cultural and linguistic skills to translate the questionnaire into appropriate versions of the target language when necessary" (Douglas & Craig, 2007, p. 33).
The main forms of collaborative work that have been proposed for questionnaire translation are the committee-based approach on the one hand, and the expert team-based approach on the other (Douglas & Craig, 2007).In the committee-based approach, the work is carried out together with collaborators working in a group that are physically in contact with each other.In expert team-based approaches, team members work individually rather than as a group, which can be more appropriate, for example, when the team is geographically scattered and operates as a virtual team (Douglas & Craig, 2007).
Advantages of the collaborative approach are that the discussion in the committee helps avoid individual researcher bias, and participation of the original authors in the committee discussions can help ensure the integrity of the source questionnaire (Alencar et al., 2007).However, Alencar et al. (2007) also point out that group discussions can be lengthy, and that consensus can be hard to achieve.
A collaborative approach of a translation procedure begins with a form of translation, frequently a parallel translation: two or more translators make independent, parallel translations of the questionnaire into the target language.A review meeting takes place consequently and the translators and independent reviewer(s) (that is, persons not involved in the translation process) discuss the various translation versions and decide on a version for final review.One of the reviewers can act as an adjudicator, that is, the person in charge of making the final decision about the translated version to be used.Frequently, committee-based review and adjudication are merged, depending on team expertise, schedules, and previous experience with similar surveys or languages.Alternatively, two committee rounds are used, one to review and revise the translated versions and the other one to decide whether to accept the revised version (Douglas & Craig, 2007).

The TRAPD translation approach
Among the collaborative approaches, Harkness, Dorer and Mohler (2010) described the framework for a five-step collaborative translation procedure that involves multiple levels of review and reconciliation known as Translation, Review, Adjudication, Pretesting, and Documentation (TRAPD).To ensure expected high-quality standards of translations, a questionnaire translation quality framework and related assessment procedures in terms of this framework should be designed and implemented.Harkness (2003) proposed to abandon the back-translation procedure.She argued that, instead of looking at two source language texts, it is more valuable in practical and theoretical terms to focus on producing the best possible translation, and then directly assessing the translation produced in the target language, rather than indirectly through a back-translation procedure (Harkness, 2003).
The TRAPD framework encompasses a multifaceted view, emphasizing both linguistic and sociocultural elements and addressing the issues of equivalence of meaning described above.
The TRAPD approach is displayed in Figure 1 and can be described as follows.
Figure 1: The TRAPD translation approach.Source: Harkness et al. (2010) Step 1: Translation.The source questionnaire is translated by two different translators.The translators each produce a draft version of a questionnaire translation, and attend review meetings.
Step 2: Review.In one or more review meetings between the translators and any additional members of the translation team, such as the survey researchers that will be implementing the survey in the country, differences between the translations are clarified and where possible resolved.
Step 3: Adjudication.The adjudicator makes decisions to reconcile any remaining disagreements following the preceding two steps and provides a final version of the target questionnaire to be tested.
Step 4: Pretesting.The questionnaire is pretested -often in a 'pilot study' or 'dress-rehearsal' -to identify problems with translated versions, in terms of difficulties in answering the questions, and to identify other problems that could impede comparability.Suggestions for modifications to the target questionnaire can arise in this step, and are assessed by the adjudicator.Transversal activity: Documentation.Qualitative and quantitative data (notes, minutes from meetings, etc.) are collected at each stage of the process and made available to support the work of persons involved in the process (translators, adjudicators) for questionnaire modification and improvement and for assessment purposes.

Survey design and translation approach in Eurofound surveys
The translation approach in Eurofound has been developed incrementally in the translation processes of the three repeated Europe-wide cross-sectional surveys Eurofound regularly carries out. 2 The general characteristics of these surveys in terms of survey design, organisational set-up of the implementation, and approach to survey translation are discussed in the following sections, followed by a brief discussion of the arguments for moving away from the back-translation-based approach.

Survey design
Prior to the ECS 2013, Eurofound had carried out five editions of the European Working Conditions Survey (EWCS, in 1990(EWCS, in , 1995(EWCS, in , 2000(EWCS, in , 2005(EWCS, in , and 2010), three editions of the European Quality of Life Survey (EQLS, in 2003(EQLS, in , 2007(EQLS, in , and 2012)), and two editions of the ECS (in 2004 and 2009).The three Eurofound surveys share the following features that bear relevance for questionnaire translation: • Coverage of all EU Member States as well as European Free Trade Association (EFTA) countries (Iceland, Liechtenstein, Norway, Switzerland) and some neighbouring countries (e.g.Turkey); • Questionnaires covering a wide range of topics designed to meet European policy needs, built on an inter-disciplinary scientific basis, and developed in close cooperation with Eurofound's tripartite stakeholders (representatives of EU Member States' governments, trade unions, and employers' organisations) and (national) experts in the field; • Increasingly elaborate strategy for quality assurance, identifying multiple quality indicators, and setting explicit targets for all stages of the survey cycle; • Increasingly high level of transparency: publishing the micro-data as well as methodological reports (including a translation report).

Organisational structure
For all its surveys, Eurofound is responsible for the survey design, the quality assurance strategy, and questionnaire development.The preparation and implementation of fieldwork -including most of the work on questionnaire translation -is contracted out.The contractor coordinates a network of local agencies in all the countries covered by the survey.Consequently, local agencies might differ, amongst other things, in their approach to survey implementation (including questionnaire translation), in their level of commitment to the contract, and the extent to which they can be controlled by the coordinating contractor.This organisational structure has implications for the decisions in the survey design, striking the appropriate balance between harmonisation, on the one hand, and functional equivalence (e.g.allowing for local variation), on the other hand.Mismatches between what is prescribed centrally and local common practice may jeopardise comparability and bear the risk of a higher price being charged, because the local partner is forced to do something out of the ordinary, and they may also lead to reduced quality, because the local agency is not willing or able to comply with the central requirements. 3These risks need to be assessed against the potential benefits of enforcing harmonisation.In drafting the contract and in designing a quality assurance strategy, Eurofound therefore must balance between prescribing what they consider 'best practice' and requesting tenderers to suggest what works best in each of the countries.

Translation approach in Eurofound surveys prior to the ECS 2013
Eurofound's approach to translation has evolved from completely outsourcing translation to a collaborative effort between the fieldwork contractor, Eurofound staff, and experts from Eurofound's network of observatories.The following process was applied in the EWCS 2010 4 and the EQLS 2012 5 , which were the two last Eurofound surveys prior to the ECS 2013 6 : • For each target language, two translators (native speakers, proficient in English and with experience in translating social survey questionnaires) each performed an independent translation of the source questionnaire; • A third translator then combined the two versions into one; • A fourth translator (not familiar with the material) then translated the questionnaire back into English (back-translation); • The results of the back-translation were reviewed centrally; • The local agencies (often the 'third translator') modified the target questionnaires based on the feedback from the coordinating contractor following the back-translation; • Finally, the coordinating contractor approved all final target-language translations, when they felt their feedback was sufficiently implemented.
Because the results of the analysis of previously collected data had raised questions about the quality of the translations, Eurofound introduced a validation stage to the translation of the EWCS 2010 questionnaire, carried out by the correspondents from Eurofound's European Working Conditions Observatory (EWCO), a network of national experts in the area of working conditions.For the EQLS 2012, a similar validation exercise was carried out in-house, by survey researchers within Eurofound.
These reviews exposed a range of issues with the translations.Some of the issues were substantive, where the translated question had a (very) different meaning from the source question; others were of a technical nature (e.g.typos, missing words).The quality assurance through back-translation therefore appeared to fall short in these two respects.On the one hand, it failed to identify differences in meaning that arose due to translation, and on the other hand, it did not ensure a sufficiently thorough review of the target text to pick up material errors (such as typos and omissions) in text processing.
These findings triggered a rethinking in Eurofound of the translation procedure that was applied.

Moving away from the back-translation-based approach
The back-translation-based approach had been developed incrementally, building from the approaches that were commonly used by various fieldwork contractors and Eurofound's own experiences and expertise.Arguments in favour of continuing the use of back-translation centred on the language capacity of Eurofound and the coordinating contractor, and the level of engagement and familiarity of the local agencies with the survey process, and with the survey instrument.It was considered risky to fully surrender control over the checking of the translated versions of the questionnaires to the local agencies.Back-translation allowed for control at the central level, despite the lack of command over many of the languages included in the survey.Arguably, in the light of the literature and the findings of the expert reviews, back-translation provided a false sense of security in the knowledge that all the local versions were checked centrally.
Ultimately, the findings from the validation reviews of the EWCS 2010 and EQLS 2012 questionnaire translations, increasing awareness of the literature that challenged the effectiveness of back-translation, and a general movement towards a more prescriptive approach to survey procurement and contract management in Eurofound, led to the adoption of a modified TRAPD-based approach for the ECS 2013.

Questionnaire translation in the ECS 2013
The European Company Survey (ECS) 2013 was based on telephone interviews with a management respondent and with an employee representative respondent.Separate source questionnaires were developed for each of these two types of respondent.The organisational structure of the survey was largely similar to the structure in previous surveys discussed above, and was characterised by multiple layers of decision-making.The survey was designed to ensure the collection of comparable data in all European Member States as well as the Former Yugoslav Republic of Macedonia (FYROM), Montenegro, Iceland, and Turkey (Eurofound, 2013a).
The questionnaires aim to capture workplace practices with regard to work organisation, human resource management, and direct and indirect employee participation as well as some respondent attitudes. 7 The source questionnaires were drafted in English.The survey was fielded in 43 languages or language variants (i.e.German spoken in Austria or French spoken in Belgium).For 34 languages or language variants independent translations were carried out.For another nine language variants (i.e.Hungarian spoken in Slovakia or Swedish in Finland) no independent translations were carried out, but adaptations were made, based on the independent translation carried out in another country (see Table 1

below).
7 Questionnaire items range between relatively straightforward questions (for instance, 'Could you please tell me for this establishment, the number or percentage of employees, who are female?'Response: [#/%]), to questions that are quite heavy on jargon or technical language (for instance, 'Does this establishment use information systems to minimize supplies or workin-process?These are sometimes known as just-in-time or lean production systems or as working according to a zero buffer principle?' Response: [Yes/No]), and questions that require adaptation to the local institutional setting (for instance, 'Which of the following forms of official employee representation currently exist in your establishment?Response: 'Do you have... [followed by country-specific lists of possible bodies of employee representation; Yes/No]').The questionnaires and their translations can be found at: https://www.eurofound.europa.eu/surveys/european-company-surveys/european-company-survey-2013/ecs-2013questionnaire/ecs-2013-questionnaire-translation.The reasons for not carrying out a full independent translation had to do with the size of the target population of the target population (e.g.Finnish in Sweden, Hungarian in Slovakia), perceived limited differences between the language variants across the countries (e.g.Russian in Estonia, Latvia and Lithuania, English in the UK, Ireland and Malta), or a combination of both arguments (French and German in Luxembourg).The process of adaptation differed considerably from the general translation approach and it is outside of the scope of this paper to be discussed in detail, hence the missing reference to it in Figure 2. The discussion below will therefore focus exclusively on the 34 languages or language variants for which independent translations were carried out.

Translation approach in the ECS 2013
Eurofound modified the TRAPD approach in order to apply it faster and to limit costs.The resulting multi-step procedure is illustrated below in Figure 2 and the process can be described as follows (also see Eurofound, 2013b).

1: Advance translation and cognitive pre-test of the source questionnaires.
The source questionnaires were translated by two translatorswith knowledge of the substantive areas covered in the survey -into German and French with the aim of detecting difficulties with translatability (advance translation).The results of the advance translation were discussed in a meeting attended by representatives from Eurofound and the coordinating contractor, the two advance translators and a survey questionnaire translation expert.The discussion resulted in an adjusted source questionnaire, adjusted target questionnaires in French and German (for the purpose of the cognitive test), and a first draft of translation instructions (item-by-item clarifications and suggestions indicating what is meant by the questions, and what meaning needs to be conveyed in their translation).The questionnaires were subsequently pre-tested in Germany, France, and Ireland through cognitive interviews to identify difficulties in understanding and answering the questions and to identify problems that could impede comparability.The cognitive interviews were carried out in two phases.The first phase consisted of conducting the full interview, while explicitly inviting questions and comments from the respondents.In the second phase, a subset of the questions was asked again, and respondents were encouraged to explicitly outline how they understood the questions and how they arrived at their answers.
The selection of questions for the second phase was based on anticipated difficulty or cultural ambiguity of as well as the position and function of the question in the questionnaire (e.g.questions in which key concepts were or routing questions).This assessment was initially made by the researchers developing the questionnaire, and amended after consultation with the coordinating contractor and with the researchers carrying out the cognitive interviews.After the cognitive interviews, the source questionnaires in English as well as the translation instructions were finalised.
Step 2: Translation after finalization of the source questionnaire.For each language, the source questionnaires were translated independently by two translators.Translators were identified by the coordinating contractor (and approved by Eurofound), and were required to have routinely carried out translations of social survey questionnaires, be native-speakers of the target language, and be proficient in English.WebTrans, which is a software tool that is specifically designed for the translation of survey questionnaires in multiple languages, was used to allow for the documentation and the sharing of translation as well as of any concerns, questions or comments for each item.WebTrans automatically ensured that elements included multiple times in the questionnaire (i.e.identical response scales, or the same expressions) appeared only once, so that these items were translated identically and reviewed in the subsequent steps.Step 3: Adjudication/review.Based on the two translations, and considering the notes prepared by the translators in Step 2, a third, synthesized version was created in a collaboration between two translators and an adjudicator who was required to meet the same selection criteria as the translators.The translators and the adjudicator participated in one or more online review meetings to decide on questions where the two translations were not in agreement.The adjudicator was responsible for the final decision.Depending on the physical proximity of the translators and adjudicator, meetings were either held face-to-face or online.To assess whether the meetings were carried out appropriately and effectively, Eurofound listened in to a selection of the online meetings and provided feedback to the coordinating contractor.
Step 4: Cross-national harmonisation.To maintain coherence across countries where variants of the same language are spoken, national adjudicators were required to share their initial national translations with adjudicators in the countries that shared their language (e.g.German in Germany and Austria). 8In cross-national harmonisation meetings, 8 Cross-national harmonisation meetings were organised regardless of whether an independent translation was carried out or whether a language variant was adapted from the independent translation from another country (except for Russian in the Baltic States); so meetings were organised to harmonise the different versions of Dutch (Belgium and the Netherlands), English adjudicators aimed to identify and address differences between the language variants that were due to translator style and were not required for reasons of cultural appropriateness.These meetings took place online, again allowing Eurofound to listen in to them.
Step 5: Review and validation.The documentation for all final target questionnaires was checked by Eurofound staff.For those languages where Eurofound staff included native speakers, the final target questionnaires themselves were checked.Where needed, adjudicators were subsequently requested to review the target questionnaire and the accompanying documentation, and to make modifications where appropriate.Wherever this review uncovered issues that suggested challenges in translating the source questionnaire, they were checked across all languages.
Step 6: Pilot interviews.Validated target questionnaires were tested in pilot interviews that were carried out to test the survey instrument on the first 50 interviews in each country.The testing of the questionnaire focused on identifying formulations that were difficult to read for interviewers or difficult to understand by respondents as well as grammatical errors and typos.Interviewers noted the presence or absence of this type of issues for each item, and based on this systematic interviewer feedback, national level fieldwork managers identified items that required a further review by the adjudicators.Where needed, adjudicators proposed final edits to the target questionnaires and provided reasons for the changes in English.Any changes required approval by Eurofound.The pilot interviews were carried out as a so-called 'soft launch', which implied that the intention was to monitor the early stage of fieldwork closely but to keep interventions in the translated questionnaire to a minimum, and where possible to retain cases (i.e.interviews) collected prior to these interventions.In practice, all the cases collected during the 'soft launch' were retained.The issues detected in the pilot interviews all had to do with the wording and routing of the source questionnaire -no issues were detected that were the results of errors in the translation.To resolve the issues that were detected, some respondents were re-contacted.In other cases, it had to be accepted that some information was missing. 9 Transversal activity: Documentation.All the steps of the translation procedure were carefully documented.Documentation produced at each step of the process was used to support the subsequent step and to improve the target questionnaires (and where necessary to modify the source questionnaire).The steps where documentation was particularly crucial were the adjudication/review and the harmonisation steps, as in these steps multiple views on the most appropriate translation had to be aligned and this process needed to be comprehensively and clearly explained in English.For the adjudication/review step, comments were provided by the team for each question and item, including how agreement was reached on the final wording and why one option was preferred over another.These comments were stored in WebTrans.Regarding the harmonisation step, the final modifications that were based on the cross-country harmonisation meetings were also commented on in the WebTrans database, with clear indication that the change was the result of this effort.

Differences with the TRAPD approach
The ECS 2013 approach differed from the original TRAPD approach in a number of ways.A first difference is the mode of the review meetings.Because these (mostly) took place online, interactions between the participants might not have flowed as naturally as would have been the case if they had (United Kingdom, Ireland and Malta), French (France, Belgium and Luxembourg), German (Germany, Austria and Luxembourg) and Swedish (Sweden and Finland). 9A report on the pilot phase was drafted, but has not been published on the Eurofound website.The report can be made available on request.
taken place face-to-face.Secondly, a cross-national review was added for languages of which slightly different versions are spoken in different countries (e.g.French in France, Belgium, and Luxembourg).Thirdly, an additional review and validation stage was included, in which native speakers checked the questionnaire wording for issues and checked the documentation for consistency and comprehensiveness.Finally, the pre-testing was not fully integrated in the translation process.The pilot test in which the questionnaires were pre-tested served more purposes than only testing the questionnaire translation and took place at a fairly late stage in the process.

Implementation of the translation approach
The success of any survey design approach relies on its consistent implementation.As we discussed before, the complexity of the organisational structure as well as the novelty of the approach to many of the key actors implementing it, meant that attention needed to be paid to the appropriate communication of the requirements and close monitoring of the implementation of all the stages of the translation process.This section outlines the various steps that were taken to ensure this.First of all, a centrally coordinated, digital environment was created in which the translation as well as all associated interactions could take place.As mentioned earlier, WebTrans was used for the translation and adjudication process.It contains a specific feature, the 'workshop mode', which was used to facilitate and document the review meetings, allowing the easy identification of items for translators to discuss.The software allowed for the audio recording of meetings and the documenting of all activities on the platform.The training of translators and adjudicators was supported by a WebEx platform, which is a software tool for holding online meetings, and which also allowed for the audio recording of the training sessions and for the use of Skype or telephone to join the conference/training when circumstances so required, facilitating maximal participation.Secondly, extensive instructions and supporting materials were drafted and distributed to guide the translation process.Prior to starting their assignments, each member of the national translation teams received training about the questionnaires, the translation process, and the tasks to be conducted.The training sessions were organised by language group and were conducted online in English.The training sessions were audio recorded for quality assurance and reference purposes.Eurofound listened in to some of them to ensure procedures were well understood and provided feedback that resulted in tweaking the training approach in subsequent sessions as well as in follow-up with attendees of previous sessions.The translators and adjudicators were provided with reference material to support and guide their work.They were provided with a glossary and detailed written guidelines.The glossary held explanations for the translation teams to support the functionally equivalent translation of key terms used in the survey instruments.Eurofound had asked experts to create short target-language vocabularies, offering translations for some crucial terms, which were recommended to be used. 10The guidelines for translators and adjudicators had the aim of giving advice on how to deal with issues, such as the appropriateness of technical terms, precision of translation, and meaning of words.Finally, the process was closely monitored.As all the translation meetings were hosted on the WebTrans platform, the coordinating contractor was able to keep close track on the timing and output of the all the meetings in all the countries at each stage of the translation process.Eurofound staff was available for support and advice throughout the process.

Efficiency and effectiveness of the process
To assess the efficiency and effectiveness of the ECS 2013 translation process, a number of sources were available.We looked (1) at several indicators in the Quality Control Plan that was compiled for the survey, (2) at the external Quality Assessment that Eurofound commissioned after the survey was completed, (3) at issues that were flagged and documented during the translation process itself and feedback that was received from the contractor, and, (4) finally, at issues that arose when analysing and reporting on the data.These sources are all based on different perspectives, the first two being more formal assessments against predefined criteria (which can be flawed in their own right) and the third and fourth being based on more subjective evaluations of those involved in the translation process on the side of Eurofound and on the side of the contractor, and of the users of the final data.Particularly these more subjective evaluations can differ considerably depending on the viewpoint that is taken.

Quality control plan
A Quality Control Plan was developed for the ECS 2013, which contained measurable indicators with explicit targets for all stages of the survey cycle (Eurofound, 2013c).The indicators were developed by Eurofound prior to survey preparation, but the exact wording of the indicators as well as the setting of the targets was finalised upon consultation with the coordinating contractor.The Quality Control Plan included 14 indicators on the translation process, covering six quality dimensions (relevance, coherence, comparability, accuracy, timeliness and punctuality, and accessibility).Nine indicators referred to aspects of the translation process ('covering skills and experience of the translators', 'quality of the training sessions', and 'comprehensiveness of documentation') and all targets on these aspects were met.One indicator referred to the timeliness of the process: the total number of days in which the process was delayed.This was reported to be 28 days.However, only in two countries the delay was 28 days, in all other countries it was 5 days or less.Nevertheless, given that a longer period was allocated to the translation process than was customary in previous Eurofound surveys, and that the process still exceeded the allocated time, the ECS 2013 TRAPD-based translation process was more time-consuming than translation processes in the past.The indicators that referred to the outcome in terms of the quality of translation show, firstly, that no item-disagreement between the translations remained unresolved after the meeting of the reviewing team; secondly, that key terms and answer categories that were repeatedly used throughout the questionnaire were not translated differently in any language, and, finally, that the pilot test did not identify any translation errors.A final outcome indicator asked for the number of discrepancies between the two initial translations, but the coordinating contractor argued this was not meaningful, as it would be extremely rare for the two translators to arrive at identical translations, so to count the instances where translations were not identical would overstate the extent to which there were issues with the source questionnaire and/or the translations.They therefore did not populate this indicator.Eurofound agreed with their assessment.Arguably, the number of discrepancies could be informative -firstly, with regard to the quality of the source questionnaire, which would be questionable for items with a relatively high number of discrepancies across all languages; and secondly, with regard to the quality of the translations, which would deserve further scrutiny for languages with a relatively high number discrepancies across all items.Rather than exploring these possible uses of the indicator, the coordinating contractor dismissed it altogether.This latter point does illustrate some shortcomings of the Quality Control Plan, as it was compiled for the ECS 2013, as a basis for the assessment of the efficiency and effectiveness of the survey process in general and the translation process in particular.It illustrates that it is not easy to formulate meaningful indicators in advance, and also underlines the importance of having high-quality indicators.That said, the Quality Control Plan showed a rather favourable result with regard to the translation process and its outcomes.The main critical point it highlighted is the negative impact of the process on the project timeline of the preparation of the ECS 2013.The time allocated for the translation process of the ECS 2013 was more generous than in previous Eurofound surveys, to allow for the more elaborate exchanges required by the team-based approach.Nevertheless, delays occurred, mainly due to the initial failure of some translators to sufficiently document the process, which is discussed in more detail below.

External data quality assessment
The external Quality Assessment of the third European Company Survey (Eurofound, 2014) included a review of the translation process used and concluded that "[t]he introduction of this translation methodology enhances the comprehension of the questionnaire and reduces the non-sampling errorsspecifically measurement errors -of the ECS, thus enhancing the accuracy of its outputs" (Eurofound, 2014, p. 23).

Issues arising during the process and feedback from the contractor
During the translation process itself, the review and validation process (Step 5) by Eurofound staff highlighted two issues.The first issue was the failure of some national teams to sufficiently document the process and to adequately illustrate the reasoning behind translation decisions.These teams were subsequently requested to provide additional documentation, which they largely followed up on.Consequently, the quality targets on the quality of the documentation were met in the end, but the process did incur delays and required additional unforeseen effort on the part of Eurofound.The second issue concerned the quality of the translations.Here as well, some of the national teams had to be requested to review their translations.Ultimately, the coordinating contractor and Eurofound were assured that the quality of the translations in all countries was good enough, but, again, delays were incurred and additional effort on the part of Eurofound and the coordinating contractor was required.When evaluating the ECS 2013 survey project, the coordinating contractor expressed a view that was less positive than the conclusion from the external quality assessment and the view of the Eurofound staff involved.They argued that the quality of the translations was not better or worse than would have been achieved using the methodology based on back-translation, but that the collaborative approach was a lot more time-and resourceintensive.They did indicate that this was largely due to the unfamiliarity of their local agencies with the collaborative approach, which prevented achieving the optimal quality and caused much of the required additional effort.

Issues arising during data analysis
During the analysis of the data and the reporting on the results some issues arose that indicated issues with translation.These issues were largely limited to items that were quite complex to begin with (for instance, where localised terms are introduced into the item based on the answers to previous items).An example is an item that asked for the number of members of the body for employee representation at the establishment.In many countries, this body can be a 'trade union delegation' or a 'works council'.In some countries, this question was relatively often answered with a very high number of members.This suggests that the localised terms that were introduced into the question referred to a 'trade union' rather than a 'trade union delegation'.As the questionnaire also includes a question on the number of trade union members in the establishment, this issue could have been picked up in the review meetings.
Nevertheless, because most questions in the ECS 2013 questionnaires were new questions -and because for existing questions, wherever possible, priority was given to existing translations -it is not possible to systematically analyse differences in the outcomes of translation between the backtranslation-based approach used in 2009 and the collaborative approach used in 2013.The assessment above is therefore largely circumstantial.It nevertheless shows that the process was largely effective in the sense that the objective of translating the source questionnaire into a large number of target questionnaires was achieved, and that only very few issues with the translations were detected, which were related to complex constructions in the source questionnaire.

Timeliness of the process and coordination efforts
In terms of efficiency, some issues were highlighted with regard to the timeliness of the process as well as with regard to the amount of coordination effort and the number of revisions required to achieve the desired outcomes.Arguably, these efficiency issues rather refer to aspects of the implementation of the translation approach (e.g.timeliness, quality of documentation, quality and training of translators) than to the approach itself.

What can be learnt from this experience: conducive and obstructive factors
Although it is not possible to systematically analyse the impact of the revised translation approach on the data quality and the differences in the outcomes of translation between the back-translation-based approach and the TRAPDbased approach as displayed above, this paper represents a first attempt to qualitatively look into and learn from the processes of the translation approach used in the ECS 2013 in order to identify conducive and obstructive factors.The aim is to formulate -based on the experience of the ECS 2013 -a number of recommendations (presented in the final section) aimed at encouraging other surveys to be equally transparent, so that these rather complex translation processes can be better understood and managed in the future.
The TRAPD-based translation procedure introduced for the ECS 2013 differed quite significantly from the process to which Eurofound, the coordinating contractor, and their local agencies were accustomed.The main differences were the more elaborate process to assess the cross-national conceptual validity, by means of the advance translation and cognitive pre-test of the source questionnaires source questionnaires, the collaborative TRAPDbased translation approach, the more extensive training of translators and adjudicators, the use of independent translations in countries sharing the same language and the organisation of cross-national harmonisation meetings,11 the more extensive documentation requirements, and the more extensive involvement of Eurofound.
Arguably, the advance translation and cognitive pre-test of the source questionnaires would have benefited any type of translation process, as it has the potential to improve the quality of the source questionnaire and generate additional information that can assist the translation process.For a collaborative approach, where the final checks are carried out on the target questionnaires and where shared understanding takes centre stage, the availability of elaborate information about what items are intended to capture is a necessity.
The other differences all refer to the process of translation and the embedded quality controls.The assessment of the coordinating contractor that the process did not achieve better translations than their regular practice highlights the challenge that was mentioned before, which has to do with the implementation of an approach that differs from the usual practice of those involved.In this case, both the coordinating contractor and their local partners not only needed to be convinced of the potential of the TRAPD-based approach, but also needed to gain experience in implementing the approach, which -as for many the ECS 2013 was the first time they engaged in a collaborative translation process -came at the expense of effectiveness and efficiency.Issues with the level of commitment and motivation were further reflected by the (initial) failure of some of the national teams to adequately document the decision-making process in cases where differences in meaning were observed between the two translations -and by the rather limited level of engagement of the coordinating contractor with the operationalisation and measurement of the quality indicators.
The requirement to use independent translations in countries sharing the same language, and the associated additional effort that was involved, triggered very different responses from the adjudicators involved.Some indicated that the additional harmonisation meetings were very useful for arriving at optimal translations.Others indicated that they felt the approach was 'over-the-top', and that the outcomes mainly depended on which adjudicator was most vocal in the harmonisation meeting.It needs to be kept in mind that cross-national harmonisation meetings only took place for four languages (Dutch, French, German, and Greek), so these observations only provide anecdotal evidence.
The more extensive documentation requirements proved to be a challenge.As outlined above, documentation provided by the local partners was not always immediately sufficiently detailed, partially due to unfamiliarity and partially due to the perception that it imposed excessive burden.On the other hand, these requirements also applied to Eurofound and the coordinating contractor, who were required to provide more extensive information about the questionnaire and the translation process.As mentioned at the start of this section, these additional efforts were conducive, particularly for the effectiveness of the translation process.
Finally, the more extensive involvement of Eurofound proved to be conducive for the effectiveness of the translation process but came at the expense of efficiency.Eurofound's involvement was largely limited to the monitoring of the quality of the process.To improve the quality of the outcomes (translations), Eurofound had to rely on the local translators and adjudicators, requiring additional time and effort on all levels.This last point touches on the core of this assessment.Because decision-making on the final outcome of the translation process is much more decentralised in a committeebased approach than in a back-translation-based approach, the effectiveness and efficiency are much more determined by the quality of the implementation process as well as the quality of the local actors involved in the process.Only if the implementation process is well organised and the local actors are wellinformed and engaged, will the translation approach be truly collaborativemeaning that a real discussion takes place at the national level that is aimed at creating the most appropriate translation -and will the benefits of a collaborative approach come to fruition.We will discuss this in more detail in the next section.

Recommendations for future surveys
A number of recommendations can be formulated based on the lessons learnt and the conducive and obstructive factors of the TRAPD-based translation approach used for the ECS 2013.They are presented in this section.1) Assess the conceptual validity and translatability of survey questionnaire.Eurofound already made efforts to assess the conceptual validity and translatability of their survey questionnaires before the ECS 2013 and has further expanded its efforts since.These efforts, specifically, the advance translation and cognitive pre-testing of source questionnaires, are required to facilitate any type of translation and particularly translation processes that rely on assessing the target questionnaires and the accompanying documentation of the translation process for quality control.
2) Motivate and facilitate the actors involved in the translation process.
The implementation of any process that involves several layers of decision-making faces challenges with aligning the incentives of the agents on each of the levels (Lafont & Martimort, 2002).A management strategy needs to be in place that ensures that all the actors involved in the translation process share an interest in, and ownership of, achieving a high-quality end-product, which are crossculturally comparable questionnaires in all survey languages.In the case of the ECS 2013, the actors were the survey sponsor (Eurofound), the coordinating team (coordinating contractor), and their national partners.To appropriately direct, control, and reward their national partners, the coordinating team needs to be highly familiar with collaborative approaches to translation and convinced of their potential advantages.Firstly, survey sponsors can assist by providing clear and elaborate information as part of the contractual arrangements.They also need to provide the appropriate means -in terms of time and budget -to carry out the translation activity.Secondly, survey sponsors face challenges with the extent to which the translators and adjudicators take ownership of, and are willing to become sufficiently familiar with, the measurement instrument.These challenges require a rethinking of the engagement with local agencies.This issue of 'buy-in' of local agencies extends beyond that of translation and can be addressed by increased involvement of the local agencies at an early stage of survey preparation.For instance, by organising a face-to-face meeting to discuss the survey design and the draft survey questionnaire early on in the process.Similarly, decisionmaking in the review meetings between the translators, between adjudicators and translators, or between adjudicators from different countries (harmonisation meetings), might benefit from being held face-to-face (Warkentin, Sayeed, & Hightower, 1997), or at least from having the participants meet face-to-face at the beginning of the process (Parke, Campbell, & Bartol, 2014).
3) Recruit translators familiar with collaborative translation approaches and surveys.Depending on budget availability and local capacity, it might be possible to recruit translators with a more tailored profile in terms of familiarity with a collaborative translation approach, and/or thorough knowledge of the survey topics.Another approach would be to involve external, local experts on the collaborative approach to translation in the translation teams, possibly as adjudicators.They could guide the translation process and ensure that it is run and documented appropriately.
4) Provide translation training.Regardless of the composition of the translation teams, translator training offers an important opportunity for the team members to familiarise themselves with the survey and the translation procedure, to create a sense of belonging to the national translation team, and to increase motivation and commitment.The translation materials need to be comprehensive yet digestible, and, ideally, training should take place face-to-face to create an interactive environment and improve translator 'buy-in' (see for instance Johnson, Aragon, Shaik, & Palma-Rivas, 2000).

Conclusion
This paper describes and assesses the translation process for a single edition of a single cross-national survey.Future research would benefit from looking at translation processes across surveys.To facilitate this, international surveys should make the organisation and outcomes of their translation processes transparent.Furthermore, more research is needed that looks explicitly at the impact of translation on measurement.A challenge in this regard is that most often approaches to translation are changed incrementally, and that in many surveys existing items are not re-translated.However, the increasing availability of metadata on translation processes in cross-national surveys should offer increased opportunities to collect and analyse this type of data.

Figure 2 :
Figure 2: Translation approach applied in the ECS 2013.Source: authors

Table 1 :
List of countries and languages in the ECS 2013