Testing a computer prototype. 

Contexts in source publication

Context 1

... architecture we would have approximately 6,000 hyperlinks on the home page, making it highly impractical to use. With the card sort approach, the number of levels in the hierarchy is kept to a minimum: breadth is greater than depth (cf., Shneiderman & Plaisant, 2004). This means that links to home pages for all major units within the School of Education are just one click away from the home page — and that most web pages are not more than two or three clicks down the hierarchy. For example, if we were to put 15 links on the home page, 20 links on each second-level page, and 20 links on each third-level page, the hierarchical table of contents could point to 6,000 unique web pages. Of course, the card sort and resulting information architecture are based on commonality of user needs, not some predetermined number of links per page. But the example illustrates that if top-level navigation pages are well-organized, a fairly large number of web pages down the hierarchy can be accessed with just a few clicks. An information architecture that results from research can be helpful when justifying the site design to administrators and faculty. The information architecture is not a matter of personal opinion or preference of any one particular person, but is based on empirical data gained through disciplined inquiry. Moreover, administrators and faculty are stakeholders too, and their needs and goals are represented in the information architecture, as well as those of other target audiences such as students, alumni and K-12 professionals. Next, the design team creates a set of paper pages, called a rapid paper prototype, containing a sample of the content and structure that is being proposed for the site (cf., Snyder, 2003). The content structure is based on the information architecture we derived from the needs assessment and the card sort. The labels on the cards become the hyperlink names, and the cards in each stack become the basis for naming hyperlinks the next level down, and the next and so on, as described above. The paper prototype is typically put into a 3-ring notebook. We write numbers or letters next to the hyperlinks (underlined text), and then we create tabbed pages with corresponding numbers or letters on the tabs, so that we can simulate web browsing. We conduct usability tests of the paper prototype by selecting members of the target audience. Usually we need to select only 4-6 members of each appropriate group (cf., Dumas & Redish, 1999; Krug, 2000; Nielsen & Landauer, 1993; Nielsen, 2000). We then observe how these people use the paper prototype to answer frequently asked questions that were identified in the needs assessment. We ask them to think aloud, record the paths they take, determine whether they find the information and find out where they would look for it in the prototype. (A very practical reference on basic principles of web design and usability testing is Krug (2000), Don’t Make Me Think!) Occasionally we alter the prototype — sometimes on the spot — and continue to test it until we have identified major problems with the design of the information architecture. If the problems are severe, we attempt to redesign the paper prototype and conduct another round of usability tests. Otherwise, we fix the problems and incorporate the design solutions in our computer prototype, which is the next phase. We do rapid computer prototyping next and conduct further usability tests. In composing the computer prototype we attend to some web elements (how users will navigate the site, approximate banner graphics, page layouts, occasional images, etc.). At this point, we are not trying to make the design look completely finished, but to get enough of it working on the web so that we can try it with users. In the past five years we have made this rapid prototyping process fairly easy for ourselves by creating approximate design templates and by using our EdWeb tools to build or re-build an existing website to create the prototype of a new or revised one. During the usability evaluation, it is our experience that participants will be more likely to notice problems and comment on them if the website looks like a prototype, not a finished product. Indeed, some intentional typographical errors, occasional missing items and crude graphics in the design prototype can help encourage users to make comments during formative evaluation (cf., Thiagarajan, Semmel & Semmel, 1974). We also seek feedback from other key stakeholders on the computer prototype at this time. Administrators and faculty tend to be more candid about design issues when they can literally see that the website is still a prototype. After modifying the initial computer prototype based on feedback from administrators and faculty, we select a new group of four to six users who are representative of each relevant target audience (cf., Krug, 2000; Nielsen, 2000), and conduct usability tests as described above for paper prototype testing (see Figure 2). At this time users try to find information under more authentic conditions than with a paper prototype. We specifically choose representative conditions, so that users are observed with PCs and Macintosh computers, typical web browsers and both broadband and dial-up connections. Users are asked to think aloud, and we record browsing paths and also use of the search engine. We use these data to identify further problems with the design, including user navigation difficulties and frustration with any web pages that take too long to display. If the problems are still severe or numerous, we will conduct new usability testing with new users after making changes in the computer prototype. When satisfied that we have fixed the big problems with the design — based on our usability findings — then we move on to the final production of the site. At this point we need to pay attention to numerous details for web publishing — i.e., getting final versions of graphics produced so that they look good and load quickly, creating and debugging cascading style sheets (CSS), making sure HTML or XHTML is valid, making each web page look good in terms of layout, checking the use of white space and the inclusion of graphics, and so on. We also need to test our hyperlinks to assure correct linkage. Normally we do these tasks in a web server folder that is hidden from the public web (i.e., nothing links to it) so that the public is unaware of the new site, but we can view it during bug testing. As the production test site nears completion, we ask key School of Education stakeholders (such as our Dean, department chairs, faculty and staff) to preview it. Based on their feedback and comments we make further cosmetic changes. We do not make major substantive changes at this time, since the information architecture and overall design have already been modified based on data from usability evaluations. As mentioned above, the reviewers at this stage know that this design process is inquiry based, and that empirical data are used to make design decisions. In fact, during the final production of the 2004 redesign, our Dean and several faculty members questioned the redundancy of audience hyperlinks on our home page. These links occur in both the horizontal navigation bar below the page banner, and again vertically in the left-most column of hyperlinks in the page body. The Dean suggested that we should consider removing those audience links in the left-most column, since it would make the appearance of the home page “cleaner” and less “busy” (and less redundant). Our response to the Dean was based on usability results. We reported the facts: none of our users during the usability tests actually used the links in the navigation bar across the top of the home page, and these links were seldom used on the second-level pages . Instead, users frequently selected the audience links on the home page in the left-hand column during our usability evaluations. Moreover, during think aloud, it was very clear why. The elaborators that are immediately after each hyperlink were frequently used to decide which category to choose. These were the empirical findings during usability tests. Why, then, did we not remove the links in the navigation bar across the top of the screen? University administrators were encouraging standardization of audience links at the top of web pages as a consistent navigation device throughout IU (e.g., see and ) and the campus web manager, speaking on their behalf, had strongly encouraged us to remain consistent with this design goal. The compromise was to leave the links in both places on the School of Education home page, even though they are redundant. If we had not had empirical data from usability testing, the links in the left column would likely have been removed, and the usability of our home page would have suffered as a consequence. Finally, when we are ready to go “live” and the Dean’s Office has approved the final design, we publish the website for the world to see (using our content management tools described below). If the website is completely new, then we need to add hyperlinks on other existing web pages on the School’s site and also notify other webmasters of the new site. If the site is a revision, usually little external change is required since we make every attempt to keep file names the same after a revision so that we do not break external hyperlinks to the site or parts of it. We also ask one of our university webmasters to make the university search engine index (or re-index) the new site immediately. After that, each page is indexed by the “spider” that follows millions of web links within the 750,000 web pages at our institution — which ordinarily occurs about once a month, and about every two weeks for pages that are updated frequently. Other search engine “spiders” (e.g., Googlebots) will also soon find and index our new site by following hyperlinks as they normally do. Then we ...

Context 2

... modifying the initial computer prototype based on feedback from administrators and faculty, we select a new group of four to six users who are representative of each relevant target audience (cf., Krug, 2000;Nielsen, 2000), and conduct usability tests as described above for paper prototype testing (see Figure 2). ...