Users were asked to describe what they saw on the screen after logging in for the first time. They were asked not to move the mouse, so that tooltips would not appear to provide clues about the icons. After about 5 minutes, they were told they could move the mouse to see the tooltips.
Several users did not know what the foot icon meant. Guesses included "the footprint of the computer", an icon related to "traveling or search and replace" or "where I've been before", and one user expected it to display "a list of tasks I've performed most recently". After moving his mouse and clicking on it, one user said he "expected 'About [Box]' type information". One user didn't think he could click on the foot because "there was no tooltip" when he moused over it. Another missed the small green arrow indicating that the foot was a menu. With a hint that the foot was the logo for GNOME, users guessed that it was a "start menu like Windows", "a 'go' button", or where they'd "click for a list of programs".
11 out of 12 users misunderstood the logout icon. They thought it was for "power saving", "to use less energy", "to set a screensaver", for "monitor settings", or "to put the monitor to sleep".
12 out of 12 users did not know what the terminal icon meant.
They referred to it as "a footprint on a monitor" or "a pathway to the system" and one said "it looks like someone has kicked the computer".
10 out of 12 users commented that the Terminal tooltip, which reads "Terminal Emulation Program", was confusing. They remarked that it was "unclear", "doesn't mean anything", and "doesn't tell me a lot". Some users still did not know what the icon or tooltip meant after clicking and opening a terminal window.
While most users recognized that clicking on the Help icon would invoke a help system, one thought that clicking it would "turn on balloon help". The more interesting feedback came when they moved the mouse over the Help icon and discussed the tooltip, which reads "Info, Man, HTML". 11 out of 12 users were confused by the Help tooltip. Several people did not know what "man" meant. People thought "HTML" meant "help with writing HTML", "help using a browser" or that there would only be help "if an item is related to the web". One person commented that the tooltip was "not intuitive" and asked "what format would I get these types of Help in?"
Most users understood the Control Center icon, refering to it as a "toolbox", "control panel" or "place to go for settings". After commenting that the Settings menu in the top menubar "would be similar to a toolbox", one user was confused about what the Control Center icon and tooltip ("GNOME Configuration Tool") were.
At least 4 people were confused by the Netscape icon. There were a couple incorrect guesses that it would invoke email. Other guesses included, "global browsing" ,"translation", and "language or keyboard language". One user who was unsure of what the icon was without using the mouse remarked, after seeing the tooltip, "This is not the standard Netscape icon, this is not helpful".
Every user commented on the "fried egg" icon which appears in the upper right corner, in the menubar. One participant asked, "Is that supposed to be recognizable?" and another said he "couldn't decipher it". Those who tried offered guesses that included, "a squashed bug", "a fish", "a crab", "a rose", "a sun" and "a spidery thing".
General comments about icon design and terminology include:
Users were asked to customize the panel by adding a clock that displayed the date as well as the time. We asked for a specific type of clock so that users would have to search for it, and in doing so interact with the system more. In the wording of the task we did not use the word "panel"; participants were simply asked to add the clock to the bottom of the screen so it appeared alongside the other icons.
There are multiple ways to add a clock to the panel:
There was a strong pattern among users' expectations of where to find clocks. Only 1 out of 12 users went directly to the panel to add a clock, adding it by right-clicking on the panel and choosing Panel > Add to panel > Applets > Clocks.
All remaining 11 users looked for a clock elsewhere. 8 of these 11 users looked in the Settings menu. One user, who received a hint while in Programs > Settings that clocks were not in the Programs menu, looked two more times in the Settings menu in the menubar before moving on. This confirms the strong expectation users held that clocks were in the Settings menu.
In addition to the Settings menu, 3 users clicked the Control Center icon, searching the Control Center for clocks until they received hints that no clocks existed in the Control Center.
Two users, guided with hints to look at the panel, were unsure of how to add a clock even as they moused over the very options that would let them complete this task. Both were guided to the panel and went to the main GNOME menu (foot) > Panel > Add to panel, but we not convinced they had found a successful way to add a clock.
It took one user, who had demonstrated familiarity with the right-mouse button in previous tasks, 10 minutes of unsuccessful searching before he thought to right click on the panel. Another user who had received two hints, had gone to Help, and had commented "at this point I'd just give up and ask a friend", said "it usually doesn't occur to me to use more than one mouse button."
Once users found the clocks they unanimously commented that the menu title "Applets" was confusing. Most said they didn't know what the term "applets" meant, and some guessed "it has something to do with Java." One user commented that the items in Applets seemed like "a bunch of random things not in one category". When asked how she would rename the Applets menu, another user offered "Accessories".
Users were surprised by the number of clocks available in the Clocks menu. They were also confused by the names of the clocks. Repeatedly, users commented that names such as "Afterstep Clock" and "JBC Binary Clock" were not meaningful or informative. Most laughed and/or were confused when they saw "Another Clock" as well as "Clock". Most arbitrarily chose "Another Clock", because they figured they already had a clock on their desktop. The clock names did not convey to the users which clock might be the correct choice.
One user also thought the menu was a way to toggle clocks on and off, because not all clocks in the menu had icons. As a result, she ended up adding multiple clocks to the panel.
Users, summarizing the experience of customizing by adding an object to the panel, said:
We asked users simply to change the font on their screen.
There are three ways to change fonts in GNOME:
In order to change all of the fonts on the screen, a user must change the fonts in all three locations.
While users looked for a place to change fonts, they commented that they were looking for "properties", "settings", or something that said "font size". The majority of users went to the Control Center on their own, several getting there through the Settings > Desktop menu. As they explored the Control Center they encountered unfamiliar terminology. There were several comments such as, "I don't know what a Sawfish Window Manager is...", "I don't know what themes are or what 'Crux' GTK+ Theme means", and "Theme Selector, I don't know what that is".
Overwhelmed by the choices in the Control Center, users needed hints to go to Sawfish > Window Manager > Appearance or Desktop > Theme Selector. One user admitted, "I never would have guessed Theme Selector for fonts." Even with hints, poor UI design and layout kept users from finding what they were looking for. One user, who spent almost 10 minutes in Sawfish Window Manager > Appearance, never saw the Browse... button to bring up the font selection dialog. Another, who received a hint said, "At this point I'm frustrated...I was in the right window and didn't even know it." A third, who went to Sawfish Window Manager > Appearance and clicked the Browse... button said, "I'm still not sure what I'll be affecting." So even when they found something that seemed like the correct choice, users were not convinced that they would be able to successfully complete the task.
Users commented that they were confused about how the font information was displayed in the Sawfish Window Manager > Appearance capplet. Users said they would rather see a button that says "Font" that the string of information in a field that, as one user pointed out "...doesn't help much, you can't even scroll through it".
Those users who attempted to change fonts in the Desktop > Theme Selector capplet were also confused by poor UI design and layout. Several found the "Use Custom Font" checkbox "annoying" and "not useful". One said it was difficult to discern if the checkbox was on or off. Two users overlooked the checkbox entirely. Their eyes went directly to the large grayed out font button and they concluded that the button "was not an option" for changing fonts because it was grayed out and they did not know how to select it. Again, a user looking at the right area of the desktop, cannot complete a task because of confusing design and layout of the UI.
After users changed the font in one location, we told them there was another place to change fonts in the Control Center. Depending on time, we either let them explore to find the other location or guided them to it.
Several users commented on the redundancy between the information in Sawfish Window Manager > Appearance and Desktop > Theme Selector.
We had them change the font in this second location and pointed out that not all the fonts on the screen had changed. We then told users that there was a third location to change fonts.
We got the following reactions:
In response to a question about how the user felt about all the choices for controlling the appearance, one user said, "There is too much...[some people] would be overwhelmed." Others answered, "There is too much stuff on the screen", "There is a lot to digest, it could be intimidating" and "I didn't know where to look." More than one user made the suggestion that some of the information "should be condensed down into an advanced section."
Users expect to go one place for fonts, make one change, and have it apply to the whole desktop. The fonts example illustrates how users will become frustrated if the underlying software does not support their mental model. If a user's mental model is that changing fonts happens magically, then we have to design to that model. We can't burden them with the distinction between fonts in the window manager, fonts in the current selected theme, and fonts inside Nautilus themes. We can't let the development process, of componentized, modularized pieces coming from different origins, permeate up to the user's level.
Despite earlier comments that some icons were not entirely intuitive, users gave generally positive feedback on GNOME's visual design:
About the experience of completing tasks in this new desktop environment, users said: