Integrated assessment of emerging science and technologies as creating learning processes among assessment communities

Integrated approaches to the assessment of technology and policy choices are found in several assessment traditions. Historically, integrated approaches have been considered as particularly appropriate for assessing complex systems that are in danger of being reduced to their composite parts, and have as such been a subject of study within systems thinking (see e.g. Smith 2010). An important motivation for developing integrated approaches has been to avoid reducing decisions with important social and environmental implications to an economic issue and such approaches have arguably been especially explored in the field of sustainability assessment, where practitioners have formed The Integrated Assessment Society (TIAS).

Recognising sustainability as a key goal of environmental management reinforces the significance of non-fragmentation and non-reduction (Bond et al. 2012). A wide range of researchers working on environmental management have contributed with important work on developing non-reductive integrated assessments over the last few decades (see for instance de Ridder et al. 2007, Van der Sluijs 2002 and Van Asselt et al. 2001). Some of these approaches are based on computational simulation models (e.g. Epstein 1999 and Hare and Deadman 2004), while others have followed a more deliberative approach (Soncini-Sessa et al. 2007 and Cohen and Neale 2006).

However, the concept of integrated assessment can be understood in various ways. Technology Assessment (TA) is another important assessment tradition that has regarded itself as having an integrating function, although the term ‘integrated assessment’ has not been a prominent concept. TA developed from decades of debate on the impacts and governance of science and technology (S&T), especially nurtured by post-war science and technology studies (STS) (van den Ende et al. 1998), and has revolved around exploring the relation between science, technology and society, including policy-making. In parallel to a critique on the limitations of expert advice and scientific reasoning in controversial and politicised science and technology issues (Wynne 1992, Jasanoff 2003), TA institutions and bodies of practice were encouraged to open up their processes to a plurality of actors and to adopt a more constructivist approach to technology assessment (e.g. Schot and Rip 1997, Guston and Sarewitz 2002). TA has therefore played a significant role in the development of participatory methods for democratic deliberation on policies dealing with the future options and risks of science and technology development (Joss and Bellucci 2002).

Other assessment traditions, or ‘advisory domains’ as we will refer to them in the following,¹ also have specific integrated approaches. For instance, impact assessments (IA), as used by the European Commission, have an integrated character where the emphasis is on causal analysis of the effects of policy interventions.² Also risk management has integrated approaches, such as integrated risk-benefit assessment.³

Emerging science and technologies (EST) appear prima facie to be in need of integrated assessment because they are often characterised by complexity, uncertainty and controversy with regard to facts and values. Emerging science and technologies are not neatly defined, but the term is usually restricted to technologies ‘that are at their early stage of development at a science and technology level’ (EC 2006, p. 13), and often includes biotechnologies, nanotechnologies, neurotechnologies and ICTs (see e.g. Robinson et al. 2013). Prima facie, arguments for integration in assessments of emerging science and technologies are related to their complex nature, having potentially significant, but to varying extents uncertain, effects in environmental, economic and social systems. Their emergent nature indicates that the uncertainties around them potentially display unknown complexity and are thus urgent to address. A reductionist or fragmented evidence base for policy- and decision-making in this field may have significant medium and long term impacts with regard to health, the environment and the economy.

Despite this situation and despite the broad variety of existing integrated approaches, there is scarce knowledge of the extent to which such technologies are being assessed in integrated ways. The questions addressed in this article are therefore to what extent emerging science and technologies are being assessed in an integrated way and, if there appears to be a need for further integration, what such integration should consist in.

We will start with a brief presentation of the methods applied in this research (Methods) and a presentation of the basis for our analysis of integration in this article (Identifying the key integration dimensions). We will then present the main findings from four case studies of assessment of specific emerging technologies analysed against this basis (Findings from the analysis of case studies). Based on the analysis of integration dimensions in the case studies, on literature studies and on dialogues with assessment professionals we will then identify integration dimensions that are currently under-developed (What kind of integration should be strengthened in EST assessment?). As a response to this gap we describe and justify an approach for further integration (The Trans Domain Technology Evaluation Process (TranSTEP)). After a discussion of the assumptions of this work and the novelty of the suggested approach (Discussion), we conclude the paper with a summary of the main points (Conclusion).

Above we have presented integration dimensions that are in need of further development. We believe that addressing the dimensions marked with ‘high priority’ in Table 2 constitutes the greatest progress beyond the state-of-the-art in integrated EST assessment. In the project it was assumed that such integration could be strengthened with a defined approach, assisting practitioners in carrying out such integrated assessments. The so-called TranSTEP approach was thus developed. This integration approach involves organising assessment dialogues across institutional and disciplinary domains; transparent, collaborative situation analysis, problem framing and method reflection; and continual process reflection to adapt to the situation under scrutiny (for details see the webpage http://transtepapproach.wordpress.com/). This includes the previously described key elements, but also includes additional elements considered useful for the approach. We will here spell out in more detail the main elements of TranSTEP⁷ (see Fig. 1).

Using the TranSTEP approach involves initiating and facilitating an assessment group composed of people from different advisory domains, as well as problem owners and other stakeholders, to integrate assessment perspectives on complex technology issues. A TranSTEP group will be convened when a problem owner identifies a specific, complex problem that needs resolution or action. This problem must be given a preliminary definition by the problem owner, allowing for establishing a TranSTEP secretariat that will assist to initially select relevant participants to the trans-domain, or TranSTEP, group. Participants in such processes can be assessment practitioners from domains such as economics, risk assessment, ethics, foresight, impact assessment or technology assessment, or from outside these domains. What domains should be involved will vary from problem to problem. In order to ensure that all relevant perspectives are brought in, problem owners and other actors should also be involved, such as representatives from industry and public research, private sector stakeholders, public sector decision-makers or administrators, NGOs or, if appropriate, the public.

The TranSTEP group will have a preliminary mandate or initial problem formulation from the initiator of the process (problem owner). Situation analysis builds on this initial formulation and it is the role of the TranSTEP group to challenge it and/or elaborate on it in close dialogue with the problem owner. Situation analysis is the first phase of any assessment (even if it is sometimes implicit) and ends up in a framing of the problem that is to be tackled by the assessment. In a trans-domain assessment process, it is particularly necessary to explicate assumptions, purposes and values and place the integrated assessment among them. This includes clearly stating the purpose to be achieved by carrying out an integrated assessment and which role the assessment aims to play. The problem will be further defined, relevant actors will be identified, perspectives and interests explored and the social and political contexts described in detail.

Where participants in a group have to explicate their assumptions, learning about themselves and about their own assumptions related to others’ is inevitable. Moreover, coming to agree on a common assessment framing necessarily involves what van de Poel and Doorn (2013, p. 123) calls ‘reflective learning’. This work is challenging and requires specific process management competencies. Bringing together such a wide range of individuals in a meta-assessment process means that it is likely that they will bring a multitude of implicit situation analyses to the table. The first task in the TranSTEP group is therefore to bring out the assumptions about the situation, critically reflect on them and agree on a common situation analysis and problem understanding that allows the group to work together.

Situation analysis should also include a preliminary reflection on what methods would be appropriate for addressing the problem framed within the group. This is necessary firstly for searching for current and available evidence that may help to address the problem, and secondly if the TranSTEP group decides that new assessments or dialogical activities are needed, since then they will also have to design such new actions. In a TranSTEP group there will be no agreed routine method to be used; the different participants may have different views on what methods would be appropriate. This is a benefit, as it allows for a transparent and reflective method discussion in the group, which in the end may yield more robust judgements than method choices based on implicit conventions or institutional traditions.

Though deliberative situation analysis, problem framing and method choice may sound like a very challenging task for a trans-domain group our experiences from organising four ‘testing workshops’ in the EST-Frame project shows that it is indeed possible. Generally, the participants in these workshops found this work hard, but fruitful.⁸

Once there is a common understanding about what characterises the issue to be assessed and there is agreement upon the problem formulation and upon suitable methods to provide knowledge on the problem, the TranSTEP group, with the assistance of the secretariat, can assess whether existing evidence (previous assessments including deliberative initiatives) can provide the foundation the group needs to conclude on the issue.⁹ The review will end up with a judgement on whether there is sufficient evidence for integrating existing knowledge into a conclusion on the problem. If the group believes that it is not, then new assessment activities will, if possible, be initiated by the group.

The group may have resources to undertake such actions themselves (such as organising a citizen’s panel) or the group may encourage relevant problem owners or stakeholders to organise such an assessment (for instance an impact assessment). In any case, the group must engage in detailed reflection on the required methods. As noted above, a wide range of methods can be used in assessments and reflecting on the strengths and weaknesses of this broad range of methods for tackling the agreed assessment problem is therefore necessary. Several former and ongoing European projects, such as DoingForesight, Sustainability A Test and Engage 2020, provide overviews of tools to make up a comprehensive tool box. These can be used to raise awareness of the wide range of method available to provide evidence for the problem solution, so that such choices are made in a reflected and transparent manner and not simply in an intuitive, implicit way resorting to default methods that might not fit the sophisticated situation analysis developed in the TranSTEP group.

Note that potential new assessment activities do not necessarily need to apply inter- or trans-disciplinary methods. What is needed may be (for instance) a traditional risk assessment, ethical assessment or foresight, if this is the knowledge lacking for an integrated conclusion in the TranSTEP group. New assessment activities can, but do not necessarily have to, be undertaken by the TranSTEP group (assisted by the secretariat). However, if outsourced, the TranSTEP group should be involved in or regularly informed about the new assessment activities and outcomes.

By drawing lessons from earlier assessments and initiating new assessments/deliberative events to fill knowledge gaps (including clarifying the extent of uncertainties that will have to be addressed by decision makers), the TranSTEP group will produce integrated conclusions to support the creation of responsible policies for research and innovation. If the group cannot initiate new assessment activities it will integrate the review into a statement of the current knowledge status, with recommendations for further assessment activities to be initiated by other relevant actors.

The TranSTEP group will decide to end the process when a) they believe there is sufficient evidence (on facts, values, perceptions or alternatives) for concluding on the issue they have defined; or b) when practical constraints (such as available funding) make it impossible to continue. At this point a report will be written integrating the results and deliberations of the process. Results integration is a matter of collective judgement in the TranSTEP group. No algorithm can be provided, only argumentation based on the preceding steps. Integration of the results will take the lessons from previous and, potentially, new assessments and apply them to the problem formulation, allowing a judgement to be taken on each aspect of the problem formulation. As such the integrated assessment will be both a meta-assessment, in the sense that it integrates the current assessment knowledge base, and a new transdisciplinary assessment. Depending on the reviewed evidence and the problem formulation the group may end up with a consensus on recommendations regarding specific decisions or policies or mapping of points of consensus and dissent. Even if the group does not end up with an agreement, reporting the situation analysis, assessment design deliberations and the type of dissent will still be of great value to policy makers and other decision makers.

It should be noted that even if Fig. 1 indicates a procedure with a clear direction, there might in reality be a need for revisiting previous stages as the group’s understanding of the issues develops. New insights might reveal the need for adjusting the initial situation analysis and framing of the issues. The procedure depicted in Fig. 1 is not intended to limit such reflective iteration. Being open to adjusting the process to new circumstances or new perspectives is an important condition for the assessment of emerging technologies in situations of complexity and uncertainty.

Moreover, it should be clear that the robustness of the conclusions of the process will depend upon the quality of the deliberations. The conclusions will mirror the process of deliberation and will be a contribution to the knowledge on the issue at hand that reflects the knowledge status at the time of integration and the composition of the TranSTEP group.¹⁰

Finally, it should be mentioned that transparency is a fundamental condition for the work in an integrated assessment process. Transparency involves being open about all issues of public interest: the situation analysis (including the problem framing), the justification of the method choices, the assessment reviews and the contested versus undisputed points of the dialogue process. Transparency is crucial in assessments that aim to give substantial advice and concrete recommendations but also in assessments that aim to explore issues in a more open fashion (see Stirling 2008) and is of particular importance of integrated assessments where the procedures by definition extends beyond the established, and often documented, conventions in the individuals domains. Revealing thoroughly the limitations and assumptions of the integrated assessment means to reveal fully the assessment as an act carried out in a specific time and place and to allow recipients to take this into account in their own reflections. By revealing the limitations of the assessment, the nature of the subsequent use of the results by others can become transparent in turn. But even if transparency is important for the legitimacy of the integrated assessment, it needs to be balanced with the need for a protected space for open dialogue.¹¹

Methodological issues of the case study and domain study research have been discussed in the articles referred to above (Forsberg et al. 2014, De Bakker et al. 2014, van Doren and Heyen 2014, van Doren et al. 2014, and Boucher et al. 2014). Although the integration analyses presented here might suffer from a certain degree of conceptual ambiguity and differing interpretations in the different case studies, the overall diagnosis of the state of integration in the four case studies are confirmed by key end-users.

It should be noted, though, that the empirical work presented here is mostly limited to the four case studies. It is possible that we would have other findings if we analysed the assessment of nano food in the UK or cloud computing in Germany. However, the analyses of advisory domains and discussions with end-users give no indication of any systematic bias resulting from the selection of case studies. Still, we believe that the next step in the research is to analyse a broader range of case studies.

The bottom-up process leading to the TranSTEP approach has had the effect that the outcome is not novel, but based on experiences of already existing practices. It must readily be admitted that similar ideas and approaches as the ones presented here have been launched before, for instance the PRIMA approach in the Dutch National Institute for Public Health and the Environment (Van Asselt et al. 2001) and the approach of the European Academy of Technology and Innovation Assessment, and in conceptual work by Tribe (1973), Schön and Rein (1994) and others referred to above. However, earlier contributions have failed to highlight the importance of targeting learning between assessment communities. This is an important point because governance of new technologies depends in great part on assessment in institutionalised communities, in particular risk assessment and economic assessment communities, and in some cases TA, ethics committees, foresight and impact assessment communities. Sometimes this work provides a transparent basis for decisions, as when risk assessment institutions assess the risk according to a regulatory framework. Other times the assessments are less transparent in their design and in the way they influence policy making. The EST-Frame end-user workshops confirmed that the assessment communities face the same challenges, especially related to dealing with uncertainty and tackling public controversy.

Moreover, advisory domains are in some cases interconnected by dependence. Uncertainties in risk assessment will often result in even larger uncertainties in the subsequent economic assessments (see for instance the common report of the EC Scientific Committees: SCCS, SCENIHR & SCHER 2013, p. 8). And decisions on risk parameters have value-dimensions that ethical committees may see as their business (see for instance WHO 2002). However, these communities – even if they recognise these dependencies (e,g, the Paris Risk Group¹²) – seldom engage with each other in practice. There might be several reasons for this; and one important reason is probably connected to the institutionalisation of the work in these domains. In order to increase integration so-called double loop learning (Schön and Rein 1994) needs to be stimulated so that the assumptions in the domains are challenged. Trans-domain assessment approaches have the potential to do this because cooperation on assessments engage domain practitioners in reflections on their own assumptions that can be a stimulus for reflections also internally in their ‘home’ communities. This can subsequently lead to domain assessments that themselves are more reflective.

Sarewitz (2010) have argued that institutional reform is needed to address the challenges for the assessment of emerging science and technologies in situations where facts are uncertain, values are disputed, stakes are high and decisions are urgent. The trans-domain nature of TranSTEP is an argument against establishing such integrated assessments as a new institution or domain in itself. The TranSTEP approach may instead be seen as an institutional innovation that continues to make each trans-domain assessment process an innovation that is adapted to the specifics of the context in which it is used. This implied institutional reform is neither hard in the sense of calling for a novel space for trans-domain assessment nor radical in the sense of abandoning existing assessment traditions, but institutional reform nevertheless.

A participant at one of the final workshops asked: “Is it revolutionary? Yes, perhaps, because the concept might consider different framings from different problem owners and therefore might be able to internalise plurality and different perspectives, which is powerful.” As such, the revolutionary aspect of trans-domain integration is not in its concepts, but in the way it might be used. Different frameworks and approaches may achieve such revolutionary effects, and TranSTEP is just one approach. The work presented here points to two strategies for further work; increased experimentation with trans-domain assessment and further research on such experiments and on the learning processes between assessors and between assessor and policy-makers.

We have here mapped out integration dimensions for EST assessment and showed how these are currently addressed in four case studies. Needs for increased integration have been discussed with reference to key contributions in the literature and discussion with end-users. From this we induced the need for an integrated approach for assessing issues in their complexity as policy problems, facilitating communication between advisory domains and integrating current assessments by applying transparent situation analysis and method choice. We have outlined these dimensions in some detail and suggested how they can be addressed in a practical process.

By taking the analytical route presented above, we have avoided two pitfalls. On the one hand, the research approach has avoided taking one given conception of ‘integrated assessment’ as authoritative, allowing instead the full range of existing contributions to the diverse field of integrative assessment to function as a reservoir for solutions to real-world quality issues in EST assessment. On the other hand, the research has also avoided the path leading towards a kind of ‘super’-assessment, which would seek to synthesise all types of ‘integration’ into a unified approach. Instead, we have sought to provide an assessment framework able to supplement and make use of existing assessment approaches so as to increase the usefulness of assessment work in general for decision-making in areas of uncertainty and contestation.

While this proposal is consistent with certain methodological proposals from the TA and sustainability assessment domains, it goes beyond these domain-specific frameworks by taking a trans-domain approach. As such, TranSTEP is integration for professionals, aiming to impact on professional practices. As a transdisciplinary and reflective approach, it is consistent with the increasing focus on RRI (Forsberg et al. 2015 and Ribeiro et al. 2016). It is useful for assessment professionals who wish to position their assessments better in the environment outside their own institutions and it is important as a learning process within and between assessment communities in the longer term. Finally, the trans-domain feature is crucial for decision makers and policy makers that need to align and balance advice from different advisory domains.