Reading Seminar 2

I have read the chapters in the 1st edition of the literature corresponding chapter 13 and 15 in the 4th edition (as that is what I have access to), that is the chapters about evaluation.

I thought the example scenarios the text brought up, about the Olympic Message System and HutchWorld, were particularly interesting as you can follow the process of evaluation in real cases. Both these cases bring up the importance of user testing and iterative evaluation. It is important to keep going back to testing; if you notice flaws and redesign, you should test again.

The text continues with going through four core evaluation paradigms: ”quick and dirty” evaluations, usability testing, field studies and predictive evaluation. The ”quick and dirty” approach is used when you want to get feedback fast. Usability tests are carried out in a controlled manner in a laboratory environment (controlled environment), as opposed to field studies which focus on the user's natural environment. Lastly, a predictive evaluation relies on experts rather than users. Of course, the best way to do and evaluation is to combine these paradigms (for example to only do a predictive evaluation would probably be useless), but how much evaluation is necessary and even possible depends on the both the project itself and budget- and time limitations.

Of course, the DECIDE-framework was also brought up. As this framework is made to help beginners in evaluating, I think it could be useful to us.

The most important part though, according to me, was the part about what, why and when to evaluate. If you don’t know what or when to evaluate something, and especially why you evaluate it, the chances are high you won’t get anything useful out of it. For example it seems like many companies think evaluation is for verification rather than the opposite, that is to find flaws that make the product unuseful to the actual users, before you let it out on the market. Because a product that isn’t useful will simply not be used.


Questions and points to discuss:

What type of evaluation paradigm should we use? It it possible for us to use more than one?
At what stage should we start evaluating (lo- or high-fi)?