Seminar 2

Chapter 13,14 and 15 (4th)

Chapter 13 

This chapter introduces evaluation and why it is integral to design processes. There are different types of evaluation that are classified into three broad categories. The first one is controlled settings involving users. The users’ activities are controlled in order to test hypotheses and measure or observe certain behaviors. The main methods are usability testing and experiments. The second category is natural settings involving users. For products that are used in public places there is little or no control of the users’ activities in order to determine how the product would be used in a real situation. The main method used is field studies. The last and third category is any setting not involving users, for example could experts identify the most obvious usability problems by criticizing, predicting, and modelling aspects of the product. The methods used includes inspections, heuristics, walkthroughs, models, and analytics.

Another issue the chapter discusses is how to inform participants about their rights and getting their consent. It is important when we start doing user evaluations that the participants have to be told what kind of task they are supposed to perform, under which conditions the derived data will be collected, and what we will use the data for when the participants have finished the task.

Chapter 14

The following chapter continues to discuss evaluation by introducing different evaluation studies that take place in various settings, from controlled settings to natural settings. The focus is on usability testing.

One central part of usability testing is collecting data about users' performance on predefined tasks. These tasks could incorporate searching for information or navigating an interface. The quantitative performance measures that are obtained produce the following types of data:

● Time to complete a task. 

● Time to complete a task after a specified time away from the product. 

● Number and type of errors per task.

● Number of errors per unit of time. 

● Number of navigations to online help or manuals. 

● Number of users making a particular error. 

● Number of users completing a task successfully.

To connect this chapter to the previous chapter I want to mention that chapter 14 has an example of usability testing, where two usability specialists conducted a usability test of the websites and apps specific to the iPad. Before the actual tests, the participants were asked to read and sign the terms and conditions of the study, which explained the following:

● what the participant would be asked to do;  

● the length of time needed for the study; 

● the compensation that would be offered for participating; 

● the participants’ right to withdraw from the study at any time;

● a promise that the person's identity would not be disclosed; and 

● an agreement that the data collected would be confidential and would not be made 

Chapter 15

The last of the three chapters introduces heuristic evaluation and walkthroughs, as ways to, in most cases, perform evaluations without an actual user. For our project, Nielsen’s 10 usability heuristics could prove very useful since we at this early stage can not perform field studies in the users' natural setting. Cognitive walkthroughs are also suitable since we rather easily can simulate our target group's problem-solving process to some degree. Pluralistic walkthroughs could also be a possibility as soon as our project has gone from a sketch/paper prototype to a more or less high-fidelity prototype developed using a software development kit (SDK).

Questions for the seminar:

Which of the types of evaluation should we use for our project? Controlled or natural settings involving users, any settings not involving users or maybe several of them?

How are we going to conduct our experiments in a reasonable fashion? Maybe an heuristic approach is suitable since it is rather difficult to try our product in a natural setting?

Would cognitive walkthroughs and predictive models such as Fitt's Law  prove useful?