Evaluation Patterns

Evaluation Patterns for Design Science Research Artefacts as described by Sonnenberg and vom Brocke

It has been criticized that current DSR processes envision the evaluation of an artefact later in the process and strictly sequence build and evaluate activities. The following DSR evaluation patterns attempt to account for the emergent nature of DSR artefacts and address this limitation. Ex-ante and ex-post evaluations are distinguished based on when they are carried out. Ex-ante evaluations are done before an artefact is built, and ex-post evaluations are done after an artefact is built.

Together, these feedback loops make up a feedback cycle that goes in the other direction than the DSR cycle. When carrying out individual evaluation activities, various evaluation methods or patterns may be utilized, depending on the context and purpose of the evaluation within the DSR process. Additionally, composite evaluation patterns could be created by combining distinct evaluation activities. The evaluation activities are highly integrated into this instance. The Action Design Research method, which uses principles to connect building and evaluation activities, is one example of such a composite pattern. This article does not discuss such composite patterns.

In order to exemplify their approach, they distinguish four evaluation types (Eval 1 to Eval 4), which are derived from typical DSR activities, and are characterized by the input and output of each activity as well as specific evaluation criteria and evaluation methods.

• Eval 1: Evaluating the problem identification: criteria include importance, novelty and feasibility.
• Eval 2: Evaluating the solution design: criteria include simplicity, clarity and consistency.
• Eval 3: Evaluating the solution instantiation: criteria include ease of use, the fidelity with real-world phenomenon and robustness.
• Eval 4: Evaluating the solution in use: criteria include effectiveness, efficiency and external consistency.