Imagining the Future as Different: Introducing a System for Participatory Evaluation

By Suzanne E. Martin

Ireland

Cite as: Martin, S. E.. (2023), "Imagining the Future as Different: Introducing a System for Participatory Evaluation", International Journal of Management and Applied Research, Vol. 10, No. 2, pp. 339-365. https://doi.org/10.18646/2056.102.23-025 | Download PDF

Abstract

Transformation of the dominant Higher Education Paradigm is necessary if design is to shape a diversified, collective ability to think and act responsibly in the world. This research paper underpins the introduction of a new Participatory Evaluation System (PES) by setting out the contextual, methodological and theoretical positioning of the assets and activities tested within the pilot. Interrogating the potential for academic evaluation as a Futures-making act, this System Design questions the purpose, positioning and potential for evaluation to become a mechanism for systemic and systematic transformation. As a proposition to actively assist the wider pedagogy, or curricula, moving and acting for adaptation, the PES employs Design Thinking practices within a Systems Thinking methodology. It imagines a capable, educational architecture to prompt discussion around how teaching and learning effectiveness is measured, what this could influence and the ways that a new model might hold space for management to anticipate change, instead of reacting to change. Systems Thinking facilitates alternative perspectives and possibilities for re-framing, not solving, problems that organisations face. Invigorating and regenerating a Design Education environment with an intervention that is holistic, effective and useful, the Participatory Evaluation System seeks to build an infrastructure that encourages staff to explore risk, and to embrace uncertain behaviour within the performative space of the present, whilst creating sustainable routes for the future.

1. Introducing the Design Contexts

This research paper presents a new Participatory Evaluation System (PES) as an emergent, agile design for evaluating and analysing the delivery of design and creative learning, and its curricula. In the dominant Higher Education Paradigm, in the situated context of the Global North, this regenerative Design (Wahl, 2016) creates an opportunity for management in academic institutions to shape a model of the organisation as a system, instead of creating models to solve problems.

The PES is understood as a model to view the education and institutional systems as they are, in and of themselves, and to utilise how they operate in practice as a way to infrastructure mediated discourse for the Futures of Design Education. The Participatory Evaluation System takes risks and embraces uncertainty in identifying innovative approaches to framing and communicating participant experiences in teaching and learning. In bringing Design Thinking practice into a Systems Thinking methodology, the Design evidences a structure for generating and applying qualitative, experiential data to transform the typology of measurement and metrics that shape a future of Design Education. The Participatory Evaluation System challenges notions of quantification – within the scope of current-state, Higher Education parameters for recognition ¬– and creates assets that are open to adaptation, appropriation and application. The Participatory Evaluation System, is one element of a new Creative Futures Pedagogical Framework (Martin, 2023), piloted during 2021/22 in the context of undergraduate studies within the School of Design at the National College of Art and Design (NCAD) as part of a multi-institutional, €10 million Irish Government funded project, the Creative Futures Academy (CFA) project.

1.1. Roadmap

In the first section of this paper, the System Design, its contexts, methodology and epistemology are positioned. Section 2 sets the scene for a different pedagogical future, presenting the theory, methods and approaches utilised in bringing the voices of participants together in shaping new, sustaining value in academic evaluation. Section 3 details the porous and flexible architecture for designing, operating and managing levels of an evaluation system by defining the Design approach. In Section 4 the Design, and its assets, are discussed in detail. Through the process of this System Design, a new set of measures are built: Section 5 points toward the potential of a new metrics, future-thinking reporting and knowledge transfer approach can have. In conclusion, Section 6 provides an overview of the preliminary impact, and rationale for future development of the Participatory Evaluation System. And Section 7 briefly sets out the next steps for the Design.

1.2. Positioning the Political, Economic Context of Design Education in Ireland

The four year CFA project seeks to act on the need for Higher Education to offer greater capacity and flexibility for Creative and Cultural sector professionals to re-skill, upskill or cross-skill, in order to address identified employment shortfalls. As part of Irish Government Human Capital Initiative (HCI) Pillar 3 funding, the CFA is an initiative in a new economic pathway, Future Jobs Ireland, launched by the Government in 2019. Aligned to this funding, the Expert Group on Future Skills Needs (EGFSN) published the Together for Design (2020) report that sets out short-term future demand for design skills and forecasts future needs from design on the island of Ireland. Summarising the typology of design skills required for Ireland’s economic success, reviewing existing skills and pointing toward drivers, policy and promotional supports that will enable design to achieve its impact, the Steering Group outline recommendations for the design community and education sector. The Design Skills Implementation Group (2021), brought together to implement the report recommendations, note in their 2021-22 Year in Review report that mainstream provision and targeted Government upskilling initiatives (such as the CFA project) embrace online, distance and blended formats therefore ensuring courses in areas of skills demand employ innovative forms of delivery. In addition, they note the criticality of developing virtual reality pedagogies, industry partnerships in course development and delivery, alongside the establishment of diverse, stackable micro-credential offerings for professionals and enterprise.

1.3. Institutional Context

Within the context of NCAD, the CFA project accelerates the development of micro-credential qualifications across all Schools, housed in a new, common 26 pathway Postgraduate Programme Architecture designed and validated during the CFA pilot phase in 2021/22. The Creative Futures Pedagogical Framework (CFP Framework), within which the Participatory Evaluation System sits, is piloted within the School of Design during 2021/22, the second year of the project and first year of delivery. The CFP Framework (Martin, 2023) is supplementary to the agreed CFA project deliverables.

The CFA project provides an external prompt and finance to amplify risky and uncertain pedagogical thinking across the institution, particularly during the pilot phase of the project. Echoing this, the CFA project reporting addresses topics such as: pedagogical innovation, academic and expanded dissemination approaches, alongside data on number increases, accessibility and movement, and impacts. New modules are piloted in Undergraduate and Postgraduate delivery across the institution: in the School of Design the CFP Framework runs in Undergraduate pilots as an approach to incubating pedagogical direction and content in Postgraduate delivery.

The pilot of this System Design, within the pilot CFP Framework, was run across two sequential academic trimesters. The learners are from one multidisciplinary, design cohort in Studio+ – an insert year, unique to NCAD, between 2nd and final, 3rd year of undergraduate studies which aims to provide professional learning experiences, research and conceptual development opportunity – that move through two synchronous, research-led critical practice courses each Trimester that equate to one, 10 ECT Thematic module vessel. This paper discusses the System Design as applied to this cohort of 32-45 learners on the sequential courses in the Thematics module, delivered by two teams of two design staff. Assessment of this module is pass/fail.

Relevant to this institutional context, evaluation processes are generally applied as a formative or summative process to understand the level of learner achievements. These are executed with accepted variation across staff bodies, years, departments and schools. The institution executes an annual student survey (as per national Higher Education Authority requirements) and hosts end-of-year student forums – facilitated by programme leads with Student Union oversight – where learners can discuss their experiences within a common, provided, discursive structure. There are periodic programme reviews, reviews at school and at institution level, all of which can involve external expert input or direction, consultation with staff and learners, with oversight of Senior Management and the office of Academic Affairs, as appropriate.

1.4. Positioning of the Design

Developed in the situated context of the Global North, the Participatory Evaluation System evidences universally applicable action for structuring mediated discourse around the future of design education. Unique to each application, specific to each particular context and the evidence created, the process has a holistic relevance for a shared future of design. In its initial scenario, the System Design navigates the anthropocentric landscape that Design and Creative Higher Education in the Global North find itself in. Providing a model for a divergent pathway, counter to the position of Design Education as an engine of the exploited and exploitative creative classes (Mould, 2018), the PES shapes strategic outputs. These products can support a theory informed, real example of a sustain-able (Fry, 2010) transformation across design Higher Education.

Building toward positioning an emergent, restorative pedagogical Framework, the PES could be described as a purposeful process for a renewed educational ecology (Siemens, 2005), holding the potential to trigger and support connected, ongoing systemic and systematic transformation within the dominant Design Education Paradigm in the Global North.

1.5. Rationale of the Design

The purpose, positioning and act of evaluating design or creative learning isn’t necessarily seen as problematic. Beyond discussion around parity, effectiveness and techniques, evaluation is seen to deliver what is asked of it. Being intrinsically linked to assessment, it predominantly serves only the current state; it is rarely tasked with a remit broader than the evaluation of the learner and their learning. It does not actively assist management approaches to the wider pedagogy, or curricula, moving or acting for adaptation.

The Participatory Evaluation System is collaboratively managed by everybody involved in its interconnected processes during project-time (Huybrechts, 2014); providing tools and assets that encourage reflective analysis within the evaluative process, all participants directly or indirectly engage with the direction of the system outputs. Placing value on distinct perspectives within the context of the whole system, a reflexive, individual approach (Adams et al., 2017) supports the requisite variety required for the model to be successful at every level. In the pilot, participants and stakeholders (or project owners) engage with the System in varying levels of proximity and influence (Figure 1). System owners sit internally in the outer ring of influence, indirectly framing knowledge used and actions undertaken within the system application. Participants range across the internal and external aspects of the system in use, with their relationships in constant flow.

Relationship Model

Figure 1: Relationship Model. Depending on what worldview is taken, the transformation experienced by each participant in the system will differ. It’s critical to note that customer and actor roles are interchangeable as the view changes, the mutability of these roles is critical to participation in, and the impact of, this system.

Managing these relationships enables distinct perspectives, needs and knowledge to be integrated effectively in shaping both the experience of using the System and to develop collaborative enquiry amongst participants. The sequence of activities and elements within the act of using the system naturally evidences conflicting perspectives. However within the cyclical structure of the System Design, divergent views converge into productive multi-voiced accounts (Roth and Kleiner, 1995) in a process that re-models the Design Thinking mechanism – of convergent-divergent thinking – into a cohesive flow of diverse voices enabled to become audible whilst remaining distinct. System outputs move through a reflective, learning cycle that continuously manages the ongoing situations, perspectives and accounts.

The ambition of this Design is surfaced by a Strategic Options Development and Analysis (SODA) Cognitive Mapping activity (Figure 2). The Design goal is to evidence the value of teaching and learning, as opposed to looking at learners and their outputs as an indication of best practice. This self-reflective mapping activity informs the design agenda for understanding and capturing deeper knowledge of what evidence could look like and what it might say about the curriculum experience. The Participatory Evaluation System is presented in a Design Mode, to model the potential, ideal version of what it might do within a creative Higher Education setting.

Relationship Model

Figure 2: Strategic Options Development + Analysis. Self-reflective Cognitive Mapping of the problem landscape to anticipate what routes might be followed in developing the Participatory Evaluation System pilot. Problems shown as bi-polar constructs.

1.6. Design Methodology

Embracing Systems Thinking as a methodology to provide a different perspective on the relatively standardised academic practice of performing evaluation, this research brings together Design Thinking in relation to Systems Thinking. This relationship is applied to a new Participatory model for curriculum evaluation.

The connected problems with Design, rooted in the structures that incubate it, are associated with language and value systems, its structures and its guiding principles, but these are ill defined and clumsily questioned. Design Thinking informs the Systems Thinking methodology applied within this Design; both methods recognise the role of interrelationships and the dynamic flow of knowledge, people and elements within any situation. Applied to the context of evaluating design and creative learning, utilising Design Thinking practices within a Systems Thinking methodology creates capability to manage and measure iterative, non-linear processes in a designerly way. In doing so, collaborative and adaptive techniques can be embedded to structure the problem definition and ideation approaches needed to better question the future of Design Education.

This paper presents a Design that reimagines the architecture of analysing curriculum delivery, pedagogy, the learning experience and the learner. Generating a cohesive package of applicable knowledge, this System Design is capable of pedagogical transformation by altering the language of understanding and measuring quality, which in turn, helps to reimagine organisational practices and the values within Design and Creative Higher Education. This dialogue then has potential to influence funding policy and thereby national institutional directives. In doing so, the Design builds knowledge from multiple perspectives. The PES design is shown as a Viable Systems Model (VSM) (Figures 3. And 4.), and employs SODA as a method for presenting individual interpretations of reality in a situation, at a macro level, within the system.

Systems Thinking embraces ambiguity in bringing together multiple perspectives and managing their interrelationships. In this paper, the system is understood as a design, as a strategic intervention in an organisation that transforms it from an existing situation to a newly constructed situation. Applied to the situation of academic evaluation, this System Design encourages risk and uncertainty through the introduction of participatory and reflexive approaches that align the contextual knowledge developed through a learning experience with academic insight.

In executing the System Design, a Multiple and Mixed Method research strategy is used; a range of methods and practices are exercised within the System activities. In analysing the application of the System, an adapted Constructivist Grounded Theory (Glaser, 2007) disposition allows system activity, experiences and supporting research to iteratively build the theoretical positioning of the Participatory Evaluation System as a new model. Led by the Academic Lead with relevant actors (Figure 1) methods applied within the System elements include: Discursive and Dialogic Design (Tharp and Tharp, 2018), Reflective practice in support of Autoethnographic Design (Schouwenberg and Kaethler, 2021) Storytelling (Lupton, 2017), Action Research and a range of Service and Participatory Design tools.

1.7. Philosophical Positioning

A Social Constructivist position sets the System Design as an Interpretivist device for constructing scenarios that bring participants together in respect of their educational environment, and context of evaluating, to grow knowledge through their interaction in relation to activities, material, each other and their environmental influences. Using Connectivism Learning Theory (Seimens, 2005) within a Systems Thinking informed axiology enhances how relationships are built, maintained and grow within the System Design that connects a learner’s experience to policy influence. Framing the interplay between the teaching and learning space and its participants, the Design supports meaning-making between the subjects and the digital/experiential/analogue objects that present themselves during the evaluation process. In this, the Participatory Evaluation System shapes an Interpretive narrative that embraces the social, the risky and uncertain as a part of the process of undertaking evaluation.

2. Positioning – The Design Education Imaginary

This Design highlights the need for connecting imagination and visualisation to any designs for systems that bridge to the future. It suggests that a shift in positioning requires consideration of meaning-making, sense-making and participation.

From a position of calibrating a vision for the dominant Design Education Paradigm, it is proposed that the future of design learning asks for imaginings and ideation (Candy and Potter, 2019), more than archetypal solutions. In a Design Education Imaginary (Gilbert and Lennon, 2005) future design learning capability will require more than a universal application of traditional deficit-led systems and management thinking (Cooperrider and Whitney, 2002). Imagining the future isn’t about fixing problems, proposals will support a re-viewing. This moment, this break-point (Manzini, 2021) between now and next, is not a problem to be solved (Cooperrider and Whitney, 2002). The Design School, as the signature pedagogy (Tovey, 2015) of Design Education, holds potential to shape strengths-based pedagogical proposals.

It is paramount that agile interpretations of how to think design, and experimental applications of doing Design Thinking, are introduced to position and direct lasting transformation. A different future for design will not be realised by updating content within a curriculum, but with a fundamental shift in the thinking that holds it along with a deeper investigation of critical questions around how to imagine that future (Scupelli et al., 2018).

2.1. Setting the Scene for a Different Pedagogical Future

To shift from paradigm to imaginary requires facilitation, mediation and brokerage for, into and through design (Frayling, 1993). Anticipating Design Learning as a site of transformation, without or beyond current discipline, is to move it from its current-state of comfort in the modern conversation, to a decentred position as a critical social theory space (Escobar, 2018) for the future. With that radically contextual driver, design pedagogy can embody a participatory infrastructure for collaboration with an architecture that promotes user-centeredness, both for its community and in service of a wider global call.

Institutional and individual models for design learning delivery, evaluation, quality assurance and the knowledge work (Staron et al., 2006) required, generally rely on existing models such as those presented by Kolb (1984), Biggs (1996), and Schön (1983), amongst others because that is what has been accepted as the [Neo-liberal] standard. Often, teaching for change, for the pedagogical transformation this paper points to, can be difficult within the existing, everyday parameters of a Design School or the context of the bigger institutions they sit within. Emancipation of knowledge is a necessary next step (Berardi, 2017) for opening up the dominant Higher Education paradigm to an equitable future design narrative.

2.2. Making meaning, Participation and Sense-making

Within this research, it is understood that all participants – staff, developers, administrators and students – are learning. In the context of design education, aligned to Connectivism Learning Theory (Seimens, 2005), meaning-making, participation and sense-making influence the process of learning, therefore in discussing any new approach to evaluating learning, these must be considered.

2.2.1. Meaning Making

Meaning is created through dimensions of social (other people and society), psychological (the self or inner psyche), emotional (feelings), sensorial/bodily and cognitive, and being (Jackson, 2011). If the development of learning, within the setting of Design Education, involves personal change, it is pertinent to look at theorist Colin Beard’s (2010) Holistic Learning Model. Modelled as an inner world, an outer world, and a sensory interface acting as the juncture for a participant registering how to receive any learning experience. In the sensory dimension, it is thought that form and tangibility are critical to understanding communication and developing the sense of self-awareness. Being actively present in a situation requires all those involved to have equal self-awareness of their interactions with that social or material environment (Illeris, 2002). In thinking toward approaches to transforming design and creative Higher Education, including devices that shape and disseminate this awareness throughout the experience, as well as in culmination, presents opportunity for expanding comprehension of what evaluation means.

2.2.2. Participatory Design

Actively engaging in teaching and learning can be considered as being part of a Participatory Design (PD) social process. PD can be understood as a risky process (Huybrechts, 2014), that asks participants to trade in previously held knowledge, to invest in the unknown outcome of group activity, with the trust that this prior knowing will benefit from, and provide benefits to, the product created. Design studio learning is a form of participation, a stage for restorative learning (Martin, 2022b) within a group act of designing yet-to-be scenarios. In that sense, the design studio, as a phenomenon (Tovey, 2015) of design education, aligns with a Participatory Futures approach (Ollenberg, 2019) in that it aims to empower somebody to shape their own future.

To evidence a different story, design teaching and learning can be viewed as a future-making act. Utilising the open-endedness of the design studio pedagogy can interrupt current patterns encouraging speculation through visions, visuals and staging actions (Candy and Dunagan, 2017; Grunwald, 2014; Inayatullah, 2008). With this, the design school studio might act as a critical, emancipatory space (Seefried, 2014) for the wider knowledge creation that Berardi (2017) seeks.

To tell future stories, Participatory Futures Research (Ollenberg, 2019) could be applied as a normative and dialogical process that facilitates social transformation; interactively exploring the present to deconstruct current concepts through which it positions the designer as a facilitator of change. This process suggests how the Systems Thinking in Practice (STIP) entity of ‘change agents’ might move toward a more integrated role of mediation and of holding space for change within a transformational design context.

The significance of multiple perspectives coming together in a problem space, as promoted by Participatory Design, is echoed in Systems Thinking. In discussing the approach to Interactive Planning, operations theorist Russell Ackoff (1979) describes the importance of involving all participants in the system and the consideration of the entire system, in all its parts, simultaneously. Advocating for interdisciplinarity and inclusion of those being served, alongside those in service of a system, Ackoff states that the success of any re-design requires commitment from stakeholders, which requires their participation in the process.

In participating, a learner employs self-awareness in order to learn to belong in that scenario and situation. Sensory awareness is critical in connecting the inner-world of a learner to the outer-world of the learning experience, however, languaging experiences and learning is problematic (Sheets-Johnstone, 2009). Bringing what is sensed into discussion is often difficult to verbalise; thoughts are mostly unconscious, and reason is emotionally charged (Lakoff and Johnson, 1999). The Holistic Learning Model (Beard, 2010) emphasises the criticality of tangibility in these dimensions for communicating what is meant by the sense of being and of learning.

2.2.3. Sense-Making

Emphasising that relationships between participants (or things) are a way to make sense of any System Design, systems practitioner Geoffrey Vickers (1894-1982) expands on the communications notion of sense-making, describing it as an Appreciative System – ongoing changes in experience and perspective inform the system, which informs actions, and consistently updates the learning of and in that system (Burt, 2010). Vickers positions systems as being devices for understanding, instead of statements for what reality should look like. He argues for a phenomenological perspective on systems thinking and the importance of considering the meanings of ‘objects’ as well as actions produced by a system.

Without entering the fully immersive and sensorial world of Abrams (1997), the idea that we can only describe the world around us by first understanding how we experience it from a phenomenological, inter-subjective awareness (Hatley, 1997) is an important consideration for any new evaluation system model.

3. Defining the System Coordination

The Participatory Evaluation System (PES) identifies methods for making sense of teaching and learning activities, and in doing so, it makes meaning for all participants, individually and as a community, through its execution. The System actively attempts to create an infrastructure that is capable of facilitating the visualisation of futures by design (Martin, 2023). Aligned to the cognitive map (Figure 2) and echoed in the Primary Operating System (Figure 3) the PES seeks to: capture teaching perspectives on delivering learning, to holistically understand the learning experience, and through that, identify whether the CFP Framework holds value as a new creative pedagogical infrastructure.

3.1. Investing in change

Introducing a suite of new evaluation approaches requires investment; it asks for a change in established, institutional methodology for measuring, as well as capturing and reporting, which in turn requires staff time, buy-in and administrative support. No organisation can justify additional costs without seeing the return. That is, a return which reaches beyond the improved epistemological or theoretical wellbeing of the disciplines and of staff. If this Evaluation System design is to be of use, and usable, within a Higher Education setting it must be accountable. The VSM, and assets, are accompanied by a breakdown of the system by-products, their features, outcomes and associated KPIs, along with a set of Value Propositions (Figure 9).

Formative model

Figure 9: Formative model of KPIs and Value Propositions for the Participatory Evaluation System. Building on the knowledge created during the pilot, a draft set of KPIs and Value Propositions can be brought forward into the next iteration of the System Design for further refinement and testing.

By looking for holistic products of the System Design that support the promotion of a culture of care (Rodgers et al., 2017) within the execution of the new evaluation approach, this System actively embeds a transformed pedagogical philosophy across the CFP Framework as a whole. It puts the stakeholders – the teachers and learners – first, equally, in the measurement and evaluation activities. The Value Propositions (Figure 9) for the Participatory Evaluation System show how human-centred mechanisms can deliver a measurable performance indicator. This signals how the values which support the system can positively impact on the day-to-day experience of teachers and learners alike, whilst generating data that validates curriculum change. In turn, this allows for the futures envisioned by all stakeholders, and captured by the System, to influence the wider Design Education Imaginary.

3.2. Complexity in Evaluation

Educator and author Peter Knight (2005) suggests that complex learning is not easily measured, that the systems must find ways to reliably and robustly foster complex achievement through radically reimagined assessment approaches. Knight advises that learning must be considered holistically and systematically; that curricula should promote self-generated theories or beliefs about ourselves, others and the difference we could make, as opposed to fitting into a formula of what complexity looks like. In this vision, learners should be enabled to validate their own knowledge, through their experiences, through a mechanism that facilitates a real transfer of knowing, instead of relying on the institution to mark achievement. To act on complexity, to shape sustainable, organisational transformation, takes more than a shift in thinking; the systems, processes, practices and mindsets must be transposed to an alternate positioning. For staff teams, departments and schools to actively engage with data – to use it as well as gather it – a dual approach of reimagining the communication, along-with the typology of data is required.

In the production of evidence that better reflects the knowledge creation capability of teaching, learning and learners (Martin, 2022a), this Participatory Evaluation System generates a different typology of data. It requires reporting mechanisms to visualise the data in a way that indicates how to support participants being confident, capable, connected, curious and committed and acting in situations with different levels of complexity (Staron et al., 2006).

4. Designing a Participatory Evaluation System

Staging and setting the opening scenes for an alternative future story in Design Education relies on an Integrational Infrastructure (Targowski, 2004) for the knowledge, information and communication infrastructures to evolve, thereby opening-up space for risk and uncertainty within the institutional cultures. In the Design Education Imaginary, the evaluation of learning and knowledge creation has new requirements. The acts of evaluation and assessment must be reimagined to provide value for the future of design pedagogy, not simply to measure against metrics or disseminate records of an institution. As a catalyst for systemic and organisational renewal, an invigorated evaluation approach might be imagined as serving the future whilst satisfying the needs of the design learning current-state.

The Participatory Evaluation System (PES) design evidences the shape of future learning based on the voices and actions of participants and stakeholders situated in the present. It acts as a mechanism for facilitating transformation by considering alternative standards of measurement and data to tell the design teaching and learning story through an experiential lens. It brings together effective, intuitive, whole-person knowledge driven by Discursive Design and narrative approaches, to build a comprehensive story of design pedagogy as it stands, facing a changing future.

4.1. Situating a New System

Formative and summative evaluation activities, when executed well, intentionally form connections between process and product, learned and applied practice, knowledge and application, and between the learner and teacher. These evaluation acts can be imagined as an improvement step, as part of an experience. In the context of this research, evaluation is a key moment of participation for all stakeholders, not just learners: where learning, drawn from being a part of the community, can be clearly seen in how it affects what that learner does with it in return (Wenger, 2000). Evaluation becomes a station in the learning journey, where the shared domain of learning, the teacher and learner’s commitment to learning, to the learning community and shared disciplinary competencies, are concretely evidenced (Lave and Wenger, 1991; Wenger 2000). Evaluation is understood as a participatory design act: engaging, connecting and providing a platform for the collective storytelling of the faceted experience of learning delivery and application, for everybody involved.

4.2. Introducing the System

Developing evaluation approaches to shape experiential epistemological knowledge creation through integrated processes, requires different supports, additional time, and a wider range of assets to be utilised within teaching. The Primary Operating System is shown as a Viable System Model (VSM), along with the first level sub-systems (Figure 3), which are fully tested within the pilot of the Creative Futures Pedagogical Framework during 2021/22 academic year. Each sub-system utilises a set of key assets which range from tools to methods, preparatory documents to guidance for acting (Figure 5). These assets integrate with teaching and delivery following a Process Map (Figure 6) which is a key management element within the over-arching CFP Framework. The VSM is modelled with four levels of recursion to indicate the potential of the System when employed in full (Figure 4).

In this System, a critical aspect is the creation of a Learning Development Guide document (Figures 6 & 7¬¬), which is issued alongside the Expression of Interest for recruiting staff to deliver the module. This document contains the sets of characteristics, critical skills, and suggested delivery methods for the module, which feature across all System Assets. The characteristics are drawn from a set of graduate creative attributes – co-created as part of the larger CFA project, in collaboration with industry and sectoral representatives – sought in cultural and creative sector professionals (Figure 8). Designing a data set that is rooted in characteristics, critical skills and methods, as opposed to levels, outputs or satisfaction, ensures that all evaluation activity focuses on the experience and meanings of teaching and learning, not on what is produced as a result. It provides an opportunity to visualise how a people-centred evaluation process might create divergent impact routes and values that reimagine future thinking, directions and moves within a Design Education institution.

4.3. A Viable System to Evaluate Delivery

The primary operating system is shown as a Viable System Model (VSM) in Design Mode (Figure 3). It is critical to understand the VSM as being more than a system of parts; it models the system as transformation processes. Each element is in dynamic interrelationship with another, following a pattern of input-transformation-process-output defined by the purpose of evidencing the value of teaching and learning (Figure 2). The VSM indicates the management elements, the flow of information and knowledge around the Evaluation process developed in the context of considering its Situation of Interest – understood as a Higher Education Art and Design institution – and the value exchange it has with its Situated Environment. The Environment includes:

  • the overarching CFA project’s funding KPIs;
  • agreed deliverables for the CFA project;
  • School of Design and Studio+ culture/approach/space;
  • learners;
  • new non-tenured staff;
  • future pedagogy and learning [of staff];
  • pedagogical research;
  • learning contents and contexts;
  • the functionality of the CFP Framework in practice.

The constituent systems (a purposeful, dynamic, input-process-output, structure) are shown in this model as a Level 1-4 Recursion (Figure 4), with detailed information (Figure 5), which indicates the variety (a measure of complexity, responding to the states of the environment that are relevant to the system viability) and ensures that the principles governing the Primary Operating System, govern the sub-systems. Developed in Design Mode, the VSM structures the ideal model for the design of a new, organised whole Participatory Evaluation System, however, activities within the pilot are located only in the Primary Operating System and Level 1 Recursion. Across the recursions, System 3 concerns System Delivery: it is where resourcing and principles for resource bargaining take place, and where decisions about the balance between autonomy and control in System 2 are decided.

In showing four levels of Recursion, the VSM indicates that any new Evaluation System considers functions at a root level, for example, gaining insights around the context for learning behaviours, attitudes and observable actions. The VSM equally ¬¬¬considers the actions and impacts at a higher level, such as ensuring programmes mesh with the value proposition of the framework and KPIs for the teaching and learning of the programmes and architecture. In considering both perspectives, the System can plot a storyline to impact on Higher Education Authority institutional performance measures and thereby influence the wider sectoral systems.

Viable System Model

Figure 3: Viable System Model (VSM). The Primary Operating System to evaluate delivery within the Creative Futures Pedagogical Framework.

Recursions of the Viable System Model

Figure 4: Recursions of the Viable System Model (VSM). The Participatory Evaluation System is shown from Levels 1 to 4 to indicate the potential impact of the Design.

Detailed breakdown of the Viable System Model

Figure 5: Detailed breakdown of the VSM Recursion Levels 1 to 4.

4.4. Participatory Evaluation System Assets

The Design can be further discussed in terms of individual assets and the role they play within the System and sub-system activities. The distribution of management throughout the system is essential to creating the requisite variety for it to work; the documents, tools, approaches and supporting knowledge (Figure 6) describe the mechanisms by which the primary activities can be coordinated in order to prevent issues arising. These are housed in System 2 across the recursions where they are utilised to introduce autonomy and control, and managed by System 3.

In the pilot, the Participatory Evaluation System is tested on new courses within existing module vessels with new, non-tenured staff. The System is designed to be implemented in full but can be introduced in packages of activity that follow the Phases of Learning segmentation (Figure 6), to enhance existing courses or modules. To provide transparency in how to manage, apply and best use the system activities, as parts of a whole, the assets are shown as aligned to three phases of the Self-Regulatory model of learning (Zimmerman, 2000), and are categorised within different Phases of Learning (Martin, 2022b). To enhance navigation, the system assets are described in terms of: objectives, ownership, function, characteristics, methods and products. All the activities or processes are developed in consideration of the design principles for scaffolding reflection and argumentation of design educational development methods (van den Akker, 1999).

Introducing any new evaluation system into existing courses might meet with staff resistance, it may not be executed appropriately or applied robustly; it is dependent on staffing, available time, knowledge and applicability. However, using the Phases of Learning segmentation as a guide, a suite of options for light, medium or intensive implementation are possible. New modules or programmes in review could benefit from applying the System in full, whereas existing modules might apply select aspects to suit independent, departmental or School needs for teaching and learning enhancement.

Participatory Evaluation System assets

Figure 6: Participatory Evaluation System assets (System 2). Assets shown in segmented aspects of delivery, and shown with owners/actors, descriptions of activity, type and indicative products of the assets.

4.5. Process Map

In deploying a new system, provision of a route-map for visualising all activity and action is vital in creating a balance between autonomy and control for participants. This Process Map resides in System 2 (Figure 3) and lays the groundwork for System 4 to function across the recursions: it plans for the identification and analysis of developments which enables the organisation to respond to changes in a timely way. In the map, the assets described in Figure 5. Can be seen in sequence, providing visibility of where issues or problems may occur and where interventions might be necessary.

Process Map

Figure 7: Process Map. Describes the sequencing of system assets and activities (see Figure 5) across the academic trimester designated for application in three functional lanes.

4.6. Creative Attributes

The Participatory Evaluation System applies the SODA method as an anticipatory design approach for route-mapping possible future directions, opportunities and needs of the system. The Creative Attributes Framework (Figure 8) characteristics act as a key for filtering directions, perspectives and individual positions across the evaluation assets and outputs. Critically, this set of characteristics, when applied within the evaluation activities, become indicators for understanding how to implement the management and delivery decisions made within System 3 of the Primary Operating System, during the pilot and beyond.

The SODA method understands that everyone comes to a situation with a different perspective informed by their individual cognitive worlds. It aims to surface those various interpretations and hold them in one place to be analysed and synthesised. Applied as an anticipatory design method, SODA guides the system by utilising the Creative Attributes Framework as a way for working with individual differences. The attributes become a key for codifying distinctions and relating them to wider overarching headlines that can be employed in shaping meaning and enabling sense-making of activity in the Participatory Evaluation System.

Creative Attributes Framework

Figure 8: Creative Attributes Framework. Co-created with project partners and industry representatives in Ireland, the attributes are categorised in four sections and accompanied by broad descriptions to help in application.

5. Reporting – Visualising Impact

The Participatory Evaluation System is an emergent and agile design for pedagogical evaluation. It develops a knowledge-sharing Systems Design approach to communicate the diverse, complex and uncertain teaching and learning experiences, containing assets for the evaluation of delivery, learning, learners and the overarching CFP Framework design itself.

Four activities are detailed in the Reflection & Meaning-making stage of the System Assets (Figure 6) which encompass both analysis and discursive modes to synthesise and communicate the system outputs. A critical, concluding exercise is the Discursive Evaluation activity. This is an element in the Design that supports staff in defining their viewpoint on the experience of teaching through a process that interrogates their thinking in relation to descriptions, comments and data from the learners. The dialogues – captured in virtual whiteboards during the facilitated, participatory activity – assemble detailed, informal dialogues that narrate the analysis of synthesised data collected from across the System assets. The conversations are guided through a series of talking points that enhance a set of data-led impact routes before focusing in on key topics that come through from learners and are echoed in staff reflective evaluation activity. Observations, supplementary information, rationale, and notes on the characteristics and behaviours which support the data are collected during the session.

Through participation in this synthesising, concluding activity, the teaching team are empowered to broker the course experience, the learning and learner evaluations, from their perspective. Critically, it provides a platform where their viewpoints can be seen, as well as holding space to reflexively process the teaching delivered, which brings complimentary value to the experience of delivering learning.

In transferring knowledge derived from the evaluation process, the Discursive Evaluation activity material is synthesised with direct learner input via the ‘Reflection Re-action’ self-evaluation activity, and insights from the weekly informal staff evaluation activity. A Bricolage method is used to capture the synthesised verbal, digital and visual dialogue from the Discursive Evaluation sessions which becomes a new kind of text-based story that uses imaginative adaptation and improvisation to shape a new cross-media communication for internal dissemination. The synthesis and coordination of System knowledge follow a similar principle to that of the Participatory Evaluation System application itself – it can be applied at an in-depth, medium, or light over-view level to suit resources and needs.

5.1. Surfacing Value

The formal evaluation report, built from applying the Participatory Evaluation System, is a design-led interpretation of the SODA method. Aggregating individual narratives and views on the delivery, learning and learners experience of the course, the report highlights: goals; key learnings; moments-that-matter; opportunities that could move forward to become affective, dynamic leverage points. As a management guide for the implementation of both the Creative Futures Pedagogical Framework, and the desirable futures the data points to, the report works with individuality and subjectivity to define capable, strengths-based transformation. Whilst SODA methodology conventionally attends to problem structuring, the Participatory Evaluation System and its report, structures the routes, actions, characteristics and storylines for capability-driven, strengths-based, organisational transformation. The reporting aims to generate knowledge that can change thinking and inspire sustainable transformation.

The Evaluation Report sets out quantified qualitative Impact Routes that are underpinned by the characteristics of the Creative Attributes Framework. The System enables common data sets to be captured across multiple activities from all participants: the Impact Routes are supported by metrics and participant impact statements. Presented as multiple, layered, stakeholder perspectives the reporting activity is able to discuss impact, values and goals from the basis of characteristics of the delivery and learning. In doing so, it employs Causal Mapping methodology to communicate these without requiring justification; the characteristics of future design learning are not a means to an end, they stage the story that Design Education needs to narrate.

Structuring opportunities in a way that deals with the reality of multiple perspectives, in the context of this system, the Creative Attributes become the component that unlocks the potential of the scenario. Utilising the Creative Attributes to take a Boundary Judgement on the system as a whole, limits scope: the Attributes lead what could be an infinite web of findings toward a concise, relevant and usable communication of data. They set out the facets of opportunity for developing the aspects of the System environment in the Situation of Interest.

6. Conclusion

If design is to responsibly act out its Futures-making role, management must be enabled to sustainably transform institutions in the dominant Design Education Paradigm. As one element in a new Creative Futures Pedagogical Framework, the Participatory Evaluation System is proposed as a model for transitioning the existing educational infrastructure toward an architecture capable of imagining and holding space to anticipate change (Bishop and Strong, 2010) instead of reacting to change. In creating a System Design as part of that Framework, discussion around evidencing the value of teaching and learning through new measurements, alongside the potential to influence the metrics that institutions and management are held to, becomes possible. Invigorating and regenerating a Design Education environment with an intervention that is holistic, effective and useful, the Participatory Evaluation System seeks to build an infrastructure that encourages staff to explore risk, and embrace uncertain behaviour within the performative space of the present, whilst creating sustainable routes for the future. Bringing together pedagogical evaluation with Value Propositions and KPIs in a System Model makes the business of auditing, cultural, it encourages discussion as well as visibility, in the construction of best practices with measurement.

Within the context of the Design Education Imaginary, new measurement of pedagogical effectiveness and learning experiences could provoke transformation across the wider Higher Education design system. The System Design presented in this paper evidences qualitative, experiential ways that design and creative learning might be quantified within the scope of current-state, Higher Education parameters for recognition. By encouraging participants to re-evaluate their co-relationships, the Design imagines a transformation model that can be experienced, evidenced and effectively managed. By exposing change to micro-adjustments through a process of rapid prototyping, the System Design imagines transformation as systemic and systematic.

Directed by each teaching and learning experience, defined by the participants and their relationships to the learning journey, the Participatory Evaluation System responds to each situated context and environment yet retains universal applicability as a management model for organisational transformation. It occupies a space between now and next, acting for complexity instead of modelling a solution. The Participatory Evaluation System has the potential to interpret and entangle (Raworth, 2017) an appropriate change infrastructure for a discordant pedagogical culture by infrastructuring mediated discourse for the Futures of Design Education.

7. Next Steps

The ambition is to apply the Participatory Evaluation System, in the new porous architecture proposed by the overarching Creative Futures Pedagogical Framework, to different [educational] environments and situations. Further testing will help to estimate the System’s impact for supporting sustainable transformation in design and creative Higher Education. Whilst this pilot and paper specifically address the Higher Education design and creative curriculum, the research is applicable to any learning organisation seeking to manage desirable, participatory futures by design.

8. References

  1. Ackoff, R. (1979), “The Future of Operational Research is Past”, The Journal of the Operational Research Society, Vol. 30, No. 2, pp. 93-104. https://doi.org/10.1057/jors.1979.22
  2. Adams, T., Ellis, C. and Jones, S. H., (2017), “Autoethnography”, In: J. Matthes, C.S. Davis and R.F. Potter (eds), The International Encyclopedia of Communication Research Methods, John Wiley & Sons. https://doi.org/10.1002/9781118901731.iecrm0011
  3. Beard, C. (2010), The Experiential Learning Toolkit: Blending Practice with Concepts. London: Kogan Page.
  4. Berardi, F. (2017), Futurability: The Age of Impotence and the Horizon of Possibility. London: Verso Books.
  5. Biggs, J.B. (1996), “Enhancing Teaching Through constructive alignment”, Higher Education, Vol. 32, No. 3, 347–364. https://doi.org/10.1007/BF00138871
  6. Bishop, P. and Strong, K., E. (2010), “Why teach the future?”, Journal of Futures Studies, Vol. 14, No. 4, pp. 99–106.
  7. Burt, G. (2010), “Learning the future faster—The role of Vickers Appreciative System”, Futures, 42, No. 4, pp. 265-270 https://doi.org/10.1016/j.futures.2009.11.012
  8. Candy, S. and Dunagan, J. (2017), “Designing an Experiential Scenario: The people who vanished”, Futures, 86, pp. 136-153. https://doi.org/10.1016/j.futures.2016.05.006
  9. Candy, S. and Potter, C. (eds.) (2019), “Design and futures”, Journal of futures studies, Vol. 1 & 2, https://fo.am/publications/design-and-futures/
  10. Cooperrider, D.L. and Whitney, D. (2002), Appreciative Inquiry: The Handbook. Euclid Ohio: Lakeshore Publishers.
  11. Design Skills Implementation Group (2021), Design Skills Implementation Group: Year in Review 2021-2022, Dublin: Department of Enterprise, Trade and Employment.
  12. Escobar, A. (2018), Designs for the Pluriverse: Radical Interdependence, Autonomy, and the Making of Worlds (New Ecologies for the Twenty-First Century). Duke University Press.
  13. The Expert Group on Future Skills Needs (EGFSN) (2020), Together For Design: Digital, Product and Strategic design Skills of the Future. Dublin: Department of Enterprise, Trade and Employment. Available at: http://www.egfsn.ie/expert-group-on-future-skill-group/all-publications/2020/together-for-design.pdf [Accessed on 29 July 2023].
  14. Frayling, C. (1993), “Research in Art and Design”, Royal College of Art Papers, Vol. 1, No. 1, 1993/4.
  15. Fry, T. (2010), Design as Politics, Oxford: Bloomsbury Publishing.
  16. Gilbert, P., and Lennon, K. (2005), Imagination and the imaginary, the world, the flesh and the subject: Continental themes in philosophy of mind and body. Edinburgh University Press. https://doi.org/10.3366/edinburgh/9780748614981.001.0001
  17. Glaser, B. G. (2007), “Constructivist Grounded Theory?”, Historical Social Research / Historische Sozialforschung. Supplement, No. 19, pp. 93–105. http://www.jstor.org/stable/40981071
  18. Grunwald, A. (2014), “Modes of orientation provided by Futures Studies: Making sense of diversity and divergence”, European Journal of Futures Research, Vol. 2, Article 30, https://doi.org/10.1007/s40309-013-0030-5
  19. Hatley, J. (1997), “The Spell of the Sensuous: Perception and Language in a More than Human World”, Environmental Ethics, Vol. 19, No. 1, pp. 109-112. https://doi.org/10.5840/enviroethics199719144
  20. Huybrechts, L., Storni, C., Lee, Y., Schepers, S., Schoffelen, J. and Dreessen, K. (2014), Participation is Risky. Approaches to Joint Creative Processes, Vol. 13, Amsterdam: Valiz.
  21. Illeris, K. (2002), The Three Dimensions of Learning, Contemporary Learning Theory in the Tension Field between the Cognitive, the Emotional and the Social. USA: Kreiger Publishing.
  22. Inayatullah, S. (2008), “Six pillars: futures thinking for transforming”, Foresight, Vol. 10, No. 1, pp. 4-21. https://doi.org/10.1108/14636680810855991
  23. Jackson, N. (2011), Learning for a complex world: A Life-wide Concept of Learning, Education and Personal Development. USA: Authorhouse.
  24. Knight, P. T. (2005), “Assessment and complex learning”, In: Proceedings of the 1st International Conference on Enhancing Teaching and Learning through Assessment. Hong Kong, 14 July.
  25. Kolb, D.A. (1984), Experiential Learning: Experience As The Source Of Learning And Development. Englewood Cliffs, NJ: Prentice Hall.
  26. Lakoff, G. and Johnnson, M. (1999), Philosophy in the Flesh. Basic Books.
  27. Lave, J. and Wenger, E. (1991), Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press.
  28. Lupton, E. (2017), Design is storytelling. USA: Cooper-Hewitt Museum.
  29. Manzini, E. (2021), Small, Local, Open and Connected. Designs For Different Futures. Philadephia Museum of Modern Art Centre, Walker Art Centre and the Art Institute of Chicago, Yale University Press.
  30. Martin, S.E. (2023), “An Instrument of Transformation – Introducing a Creative Futures Pedagogical Framework”, Journal of Futures Studies, Special Issue: The Future of Futures Thinking in Design Education, Vol. 28, No. 2, pp.
  31. Martin, S.E. (2022a), “Evaluating the Future: Capability and Knowing Through Design Learning”, In: AMPS, Transformative Teaching Education Conference, Proceeding Series 31, 15-17 November, 2022.
  32. Martin, S.E. (2022b), “Redefining the structure: A design for remote studio learning”, in:Lockton, D., Lenzi, S., Hekkert, P., Oak, A., Sádaba, J., Lloyd, P. (eds.), Design Research Society 2022, Bilbao, Spain. 25 June - 3 July, https://doi.org/10.21606/drs.2022.672
  33. Ollenberg, S. (2019), “Futures-design-process Model for Participatory Futures”, Journal of Futures Studies, Vol. 23, No. 4, pp. 51-62. https://doi.org/10.6531/JFS.201906_23(4).0006
  34. Raworth, K. (2017), Doughnut Economics: Seven Ways to Think Like a 21st Century Economist. London: Random House.
  35. Rodgers, P.A., Innella, G. and Bremner, C. (2017), “Paradoxes in Design Thinking”, The Design Journal, Vol. 20, No. 1, pp. S4444-S4458. https://doi.org/10.1080/14606925.2017.1352941
  36. Roth, G. and Kleiner, A. (1995), Learning About Organisational Learning – Creating a Learning History. MIT Centre for Organisational Learning.
  37. Schön, D. (1983), The reflective practitioner: How professionals think in action. New York; Basic Books.
  38. Schouwenberg, L. and Kaethler, M. (eds.) (2021), The Auto-ethnographic Turn in Design, Amsterdam: Valiz.
  39. Scupelli, P., Wasserman, B., Wells-Papaneck, D. and Brooks, J. (2018), The Futures of Design Pedagogy, Learning, and Education. 21st DMI: Academic Design Management Conference: Next Wave. London, UK. 1-2 August.
  40. Seefried, E. (2014), “Steering the Future. The Emergence of “Western” Futures research and its production of expertise, 1950s to early 1970s”, European Journal of Futures Research, Vol. 2, Article No. 29, https://doi.org/10.1007/s40309-013-0029-y
  41. Seimens, G. (2005), “Connectivism: A Learning Theory for the Digital Age”, International Journal of Instructional Technology and Distance Learning, Vol. 2, No. 1, https://www.itdl.org/Journal/Jan_05/article01.htm
  42. Sheets-Johnstone, M. (2009), The Corporeal Turn, an interdisciplinary Reader. Exeter: Imprint Academic.
  43. Staron, M., Jasinski, M. and Weatherly, R. (2006), Life-based Learning: A Strength-based Approach for capability Development in Vocational and technical Education. Australian Government Department for Education Science and Training and TAFE NSW. Available at: http://learningtobeprofessional.pbworks.com/f/lifebased_full_report.pdf [Accessed on 29 July 2023].
  44. Targowski, A. (2004), A Grand model of civilization, Dialogue and Universalism. In Comparative Civilizations Review, Vol. 51, No. 51, Article 7. Available at: https://scholarsarchive.byu.edu/ccr/vol51/iss51/7
  45. Tharp, B.M. and Tharp, S.M. (2018), Discursive Design: Critical, Speculative, and Alternative Things. USA: MIT Press.
  46. Tovey, M. (2015), Design Pedagogy: Developments in Art and Design Education. UK: Gower Publishing Company.
  47. Wenger, E. (2000), Communities of Practice: Learning, Meaning, and Identity. New York: Cambridge University Press.
  48. van den Akker, J. (1999), “Principles and Methods of Development Research”, In: van den Akker, J., Branch, R.M., Gustafson, K., Nieveen, N., Plomp, T. (eds), Design Approaches and Tools in Education and Training, Dordrecht: Springer, pp. 1-14.. https://doi.org/10.1007/978-94-011-4255-7_1
  49. Wahl, D. C. (2016), Designing regenerative cultures. UK: Triarchy Press.
  50. Zimmerman, B. J. (2000), “Attaining self-regulation: A social cognitive perspective”, In M. Boekaerts, P.R. Pintrich, and M. Zeidner (eds.), Handbook of Self-regulation. USA: Academic Press, pp. 13-39. https://doi.org/10.1016/B978-012109890-2/50031-7