Evaluations of GNOME Usability:
Expanding the Appeal of GNOME


Calum Benson

Sun Microsystems Ltd.
Dublin, Ireland
+353 1 819 9000

calum.benson@sun.com



ABSTRACT

In March 2001, at GUADEC in Copenhagen, we presented the results of the first major GNOME usability study.. Several usability enhancements were suggested that we felt would have to be made if GNOME was to become a credible contender in the UNIX desktop arena.


Since that paper was presented, the GNOME Usability Project (GUP) has been formed, the GNOME Human Interface Guidelines (HIG) have been published, and much work has gone into improving the usability of GNOME's subsequent releases, the most recent being GNOME 2.2.


This paper has two goals. The first is to assess the impact of that original usability study two years on, reflecting both on those recommendations that were implemented, and those that were not. The second is to consider the impact of the GUP and HIG, and what might be required of the GNOME project to ensure that GNOME usability continues to improve as we look towards GNOME 3.0 and beyond.


  1. BRIEF HISTORY OF GNOME USABILITY

It would be impossible to list the many invaluable contributions to GNOME usability made by individuals over the years. To avoid indadvertently omitting anyone from such a list, this section summarises instead some of the major collective contributions.

Andy Hertzfeld and Mike Boich, part of the original Apple Human Interface Group, formed Eazel, Inc. in 1999. Their vision was to create an easy-to-use Linux desktop that was also a gateway to networked services such as file sharing, automatic software updates and off-site file archiving. Lack of funding saw Eazel close its doors in 2001, but its legacy lives on in the shape of the Nautilus file manager, which is still at the heart of the GNOME desktop today.

Ximian, Inc. (formerly Helix Code) was founded by Nat Friedman and GNOME creator Miguel de Icaza, also in 1999. Ximian was among the first to improve both the visual design and the interaction design of its GNOME distribution to provide a more satisfying user experience. An improved menu structure and the Doorman configuration wizard made life easier for GNOME novices, the Red Carpet web updater simplified package management, and distinctive graphics from Tuomas Kuosmanen and Jakub Steiner made it among the most visually attractive desktops available to the UNIX user. Ximian continues to be a key contributor to many aspects of the GNOME desktop and its underlying technologies.

Sun Microsystems, Inc. announced its intention to support GNOME on its Solaris Operating Environment in August 2000, and one of the first tasks it undertook was to perform a usability study of GNOME 1.2. The results [1] were presented at GUADEC in Copenhagen in March 2001, prompting a commitment from GNOME project leaders to address many of the issues that it raised in GNOME’s subsequent 2.0 release.

While companies such as these were able to commit professional UI design resources to parts of the GNOME project, what was largely missing up to this point was sufficient guidance for other GNOME developers to help them write applications with a distinctive, consistent GNOME look and feel whilst also avoiding common usability problems. A previous attempt to write user interface guidelines had failed to produce anything authoritative, following disagreements among contributors. And although James Cape’s “GNOME UI Hit Squad” team [2] had successfully redesigned some elements of the desktop, it had become somewhat dormant.

Following GUADEC 2001, the GNOME Usability Project (GUP) [3], led by Seth Nickell, was formed to address these issues. Its first task was to draft user interface guidelines that would be endorsed by the GNOME community. A small team of volunteers with usability experience was assembled, and a year later the first GNOME Human Interface Guidelines [4] were released. They are now the main reference point for the comprehensive UI review that is carried out before each point release of GNOME.

  1. MARCH 2001 STUDY: REVISITED

This section of the paper reprises the key recommendations from the 2001 usability report, and indicates whether or not they were implemented. Only the recommendations themselves are quoted here; for a full account of the participants’ comments and the observed behaviour that led to these recommendations, see the original usability report .

    1. Study participants

The 2001 study involved 12 participants from business, scientific and creative fields. The group comprised both men and women, all with at least 3 years’ experience of using graphical desktop software. All had prior experience of both Macintosh and Windows desktops, and 4 had prior experience of CDE. None had used GNOME before.

The software was tested on Red Hat Linux 6.2, and comprised community GNOME 1.2 plus newer versions of Sawfish (v0.36.1) and Nautilus (v1.0). The study was conducted in Sun Microsystems’ usability labs in Menlo Park, CA, on March 13th-16th 2001.

    1. Study tasks

The participants followed a predefined list of tasks [5], which required them to log in to the system, comment on the look of the desktop, and use GNOME for some predefined file management and desktop customisation tasks with which they were expected to be familiar from their previous computer experience.

    1. Which recommendations were implemented?

      1. Architectural Issues

      1. Logging In

      1. Exploring the Desktop

      1. File Management

      1. Customisation

      1. Logging Out

      1. Summary

Of the 29 recommendations in the 2001 study report, 21 have since been partially or fully implemented—a respectable achievement. However, it would still be worth revisiting those recommendations that have not yet implemented, to understand why not.

  1. WHAT NEXT?

    1. Re-assessing the GUP and HIG

For whatever reason, motivation to fix usability bugs seems to have declined somewhat since the release of GNOME 2.0. Then, the release team sanctioned a 'UI review' to be carried out after the general UI freeze, in which every application was assessed for compliance with the Human Interface and Documentation guidelines, and for basic accessibility compliance. In the GNOME 2.0 UI review, 30 usability bugs were reported, 29 of which were addressed in time for the release.

The exercise was repeated for the GNOME 2.2 UI release. This time, 66 bugs were reported—a surprisingly large number, considering this was the first release for which the HIG was available for developers to follow throughout the entire development cycle—but only 17 were addressed.

      1. How can we improve the GUP?

One of the original hopes for the GUP, as for the UI Hit Squad before it, was that it could act as a “UI design consultancy" for the community. So far, however, it has probably not been as effective as it might have been.

Perhaps the most successful innovation has been the introduction of the usability keyword in GNOME Bugzilla, and the routine cc’ing of usability-maint on usability bugs. These have allowed members of the GUP to track and comment on a much wider range of usability bugs than before.

On the downside, weekly IRC meetings among GUP members to discuss topical usability issues were short-lived, for familiar reasons. IRC meetings are difficult to keep focused, and timezone co-ordination proved tricky.

Neither has the GUP responded particularly effectively to the occasional new developer who appears on the mailing lists with an idea for an application, who has visited the GUP website and asks for usability help. At best, they are usually advised to go and read the HIG, and perhaps post some screenshots later on for comments. A more organised GUP might perhaps, on a case-by-case basis, decide to “assign” a willing member to work with any such developers, to help them see their application through from inception to stable release.

Gathering feedback from users is another challenge. Talking about the Mozilla project, Kamiyo Nakakoji [9] says :

Reports from lots of users is unusual too; my usual rule of thumb is that only 10% of users have any idea what newsgroups are (and most of them lurk >90% of the time), and that much less than 1% of even Mozilla users ever file a bug. That would mean we don't really ever hear from 90% of users, unless we make some effort to reach them.

It’s difficult to estimate how many of GNOME’s users we hear from, but it’s fairly likely that, as with the Mozilla project, many users probably don't know how to give feedback at all. IRC is an unfamiliar environment to those who are more used to chat clients like ICQ, AIM and MSN. Bugzilla is an intimidating place to file bugs. And the list of GNOME mailing lists is long, and almost entirely technically-oriented. Rarely a day goes by without some newcomer’s question being answered with "you need to ask that on list x, this one is for y". Simply changing the name of gnome-list to something like gnome-users or gnome-support, and replacing the standard mailman ‘list of lists’ web page [10] with a more thoughtfully-constructed one could help here.

Even the feedback we do receive is currently diluted among many different forums, so more of it than we would like is probably being lost— again, though, it's hard to estimate how much. Numerous IRC channels, gnome-list, GNOME Bugzilla, Ximian Bugzilla, gnomedesktop.org, gnomesupport.org... how many busy developers have time to monitor all of those? Hopefully, the new GNOME website will go some way to re-unifying all these invaluable sources of information.

      1. How do we improve the HIG?

Most developers support the idea of a UI guidelines document for GNOME. But the figures from the most recent UI review suggest that it is not being used as effectively as we would like. Presumably, either developers are finding it too hard or time-consuming to follow, or they believe they have followed it when they haven’t, or they think they only need to fix HIG-compliance bugs later on, once they’re filed in Bugzilla. The GUP should make the effort to find out which.

Regardless of that, there are some other issues with the HIG that need to be addressed.

The original writing team has largely moved on to other things, so maintenance is patchy and little new material is being written. For the HIG to move forward again, more volunteers are probably required.

Lack of time and resources meant that the HIG was not thoroughly reviewed, even by its authors, in the weeks leading up to its 1.0 release. This task should be high on the agenda of a revitalised team.

In recent months, development of the subsequent draft version has also stalled somewhat in expectation of a merger with the KDE guidelines, a project that experienced a brief flurry of publicity [11] and has since gone rather silent again.

The HIG is also unlikely to get any shorter as it develops, particularly if it is merged with an accessibility guide, which has been mooted in the past, or with the KDE guidelines. One solution might be to go the route of Sun’s Java Look and Feel Design Guidelines, which are published in two volumes, the second of which is devoted to “Advanced Topics” [12].

    1. Conceptual models

Open source software design traditionally produces fewer formal design documents than a commercial application—none at all, in many cases. Those that are produced are often retrospective and text-only, such as API documentation. This certainly allows for the rapid design-development-feedback cycle that is the hallmark of most successful open source projects, but it bypasses a useful step that can be used to validate the usability of a design even before a low fidelity prototype is produced, and certainly before a line of code is written.

The conceptual model of an application describes its objects, and their attributes and inter-relationships. A user’s mental model describes their personal interpretation of the objects, attributes and relationships, which may well expand or change as they learn to use the application. Not surprisingly, then, the most usable software is that where the user’s mental model maps directly on to the application’s conceptual model, and continues to do so as learning more of its features causes the user's mental model to expand.

A conceptual model need not necessarily be complex. Indeed, the more complex the model, the more difficult it becomes to successfully expose it via the application’s user interface, and the less likely it will be to match the user’s mental model as they learn how to use it.

For example, an idealistic conceptual model of the GNOME desktop might look something like this (based on an original by Nils Pedersen [13]):




Figure 1 – Example conceptual model of the GNOME desktop.

In this model, a workspace contains one background, and zero or more panels. A panel can also contain zero or more other panels. Finally, a background or a panel can contain zero or more desktop objects, where a desktop object is either a menu, a launcher, an applet, or a file icon.

Even something this simple highlights possible shortcomings in the existing GNOME implementation, which could cause the user’s mental model to develop inaccurately:

By comparison, someone who is just learning to use GNOME for the first time might have a mental model1 more like this:




Figure 2 – Example mental model of the GNOME desktop for a new GNOME user.

In this model, the user is not yet aware of concepts such as workspaces, drawers or applets (other than the window list applet that appears by default—they probably haven't tried right-clicking on the panel yet), nor have they learned the difference between file icons and launcher icons. Their concept of the “desktop” (which to them is just the wallpaper plus the icons on the desktop background—it doesn't include the panel) may be shaped by previous experience with a different desktop, or by inconsistencies in the use of terminology in the UI, or even just by the words used by whoever showed them how to use GNOME.

Other than not considering the panel to be part of the desktop itself, this user's mental model is not too far away from being a subset of the conceptual model, which is good. Note however that the models may grow further apart as they discover new features such as drawers and applets, which they might consider to behave inconsistently for the reasons mentioned above.

What happens if the user does develop an inaccurate mental model? According to User Interface Engineering's Jared Spool, most likely they will either “build inefficient usage patterns into their way of working with the application”, or “abandon functionality because their mental model shows it to be unreliable or taxing”. [20]

It would be unrealistic to expect GNOME application developers to produce and maintain conceptual model documentation for all their applications. But for those that are willing to do so, having them publicly available and reviewed by the GUP prior to implementation could perhaps prove to be at least as effective as having their final user interface designs critiqued.

    1. Personas and scenarios

Most usability gurus advocate a persona-based approach to user interface design—Alan Cooper [14] is among the more vociferous. Personas are a distillation of your software’s target audience into a handful of fictional users. For example:

Joe is a 24-year-old man. He has a clerical position with the IRS in Springfield, IL for 2 years. This is his first job out of college. He has a Sociology degree. Joe is relatively comfortable with computers but is not an expert (as opposed to his older colleagues who generally distrust the computers). He knows how to do some word processing, use a browser, and IM with friends. He likes to surf the web for information on camping, snowshoeing and skiing, hunting, and his SUV.

Most of his work day involves entering and correcting demographic and financial data on people. There are a variety of custom software applications he uses to do this. Each has its own authentication scheme. As such, Joe has become adept at logging into and out of different systems. He tends to use only one or two passwords on a rotating basis so he can remember them. He does surf the web from work. A small part of this is for work to collect information from other government web sites. He surfs for info on his hobbies (listed above) during his lunchtime.

Joe has an old laptop at home. He originally got this laptop when going to college. He has a dial up account to access his personal email, and to IM with friends. He does not understand that he could do most of these tasks from work; he sees his work-computing world as a separate entity from his personal/home computing world.

Focusing on such personas, provided they are representative of your users, helps you focus on the tasks and interaction methods your application should support. Havoc Pennington has been vocal in promoting such an approach for GNOME [16], perhaps maintaining a set of GNOME personas as part of the HIG.

Personas generally feed into scenarios. A scenario is a narrative description of a task that a user (or a persona) will use your software to achieve. They focus design efforts on the user's requirements, which are distinct from technical or business requirements. They are written to be understood by people with no technical background, and are therefore suitable for use during participatory design activities—they formed the basis of the tasks used in the 2001 usability study, for example. Here is a very simple example of a scenario:

Róisín has had her office machine updated from CDE, which she has been using for years, to GNOME 2.2. To help her make the switch, she would like to add all the application icons she had on her CDE panel to her new GNOME panel.

Note that the same scenario can generate different requirements on the software, depending on the experience and expectations of the persona that is “acting” in it. Some personas might rely heavily on online help to guide them through any unfamiliar task (requirement=tooltips, Help button), while others prefer to experiment (requirement=good Cancel/Undo facilities). Likewise, some might be expert mouse users who expect drag and drop to work everywhere, while some might be forced to use a keyboard because the only space in their office for a PC is the top of a filing cabinet!2.

All relevant aspects of the interaction are included in a scenario, even where they are outside the current scope of the technology. Such references may include cultural and attitudinal issues. For example, in some scenarios, the fact that Róisín is continually interrupted by telephone calls may be just as relevant as the applications she uses.

Scenarios are easier to write for individual applications than for a whole desktop, because an application’s purpose is typically more tightly-focused. However, developing a set of GNOME desktop-wide scenarios could be useful for characterizing inconsistencies in different applications’ conceptual models, and for deciding where proposed new desktop features would most usefully be added.

    1. What are other projects doing?

When the 2001 usability study was published, usability on large open source projects was a comparatively novel concept. Since then, an increasing number of articles, academic papers and workshops on the subject have appeared, many of which reference the efforts of the GNOME project.

A review of those materials suggests there is still little in the way of tried and tested usability processes to be gleaned from other open source projects however, although some papers certainly contain some interesting ideas. Nichols and Twidale [15] for example, speculate on the possibility of packaging self-administered, computer-assisted usability tests along with development versions of applications, the results of which can be returned immediately by email. They also reference some work on tools that 'score' a UI based on metrics such as text and widget density. If these proved useful, tools could presumably be written quite easily to extract such metrics from Glade files.

At CHI 2002, some members of the GUP co-organised two 'usability in open source' sessions [16]. CHI luminary Steven Pemberton, with no knowledge of GNOME or its development processes, suggested there that the best way ahead on such projects might be to assemble a small group of usability ”consultants” to offer advice and design resources to the other teams on the project. This, of course, was exactly the rationale for creating the GUP in the first place, and the UI Hit Squad prior to that. Affirmation then that GNOME was somewhat of a pioneer in this field… in other words, we rock! ;)

    1. What about another usability study?

GNOME 2.x is now a sufficiently mature and different product from GNOME 1.x that another usability study would undoubtedly be useful. Sun Microsystems hopes to carry out one such study later in 2003, comparing the usability of GNOME more directly with other desktops than was the case in the 2001 study.

  1. SUMMARY

GNOME’s usability has improved in many ways since my first sight of version 1.2 a little over two-and-a-half years ago. Key to these improvments have been the formation of the GNOME Usability Project, the publishing of the Human Interface Guidelines, and above all the considerable efforts of developers and the release team to ensure improved levels of usability for all core GNOME applications.

In order to take the next step, however, I believe we still need to:

Improving our efforts in these areas will, I believe, provide a solid framework for continuing to improve GNOME usability as we head towards GNOME 3.0 and beyond.

  1. ACKNOWLEDGEMENTS

All product and brand names mentioned in this paper are trademarks of their respective owners.

  1. REFERENCES

  1. Smith, S. et al., “GNOME Usability Study Report”, http://developer.gnome.org/projects/gup/ut1_report/report_main.html, March 2001.

  2. Cape, J. et al., “GNOME User Interface Improvement Project”, http://web.archive.org/web/20010606111857/developer.gnome.org/gnome-ui/hitsquad/, August 1999.

  3. GNOME Usability Project, http://developer.gnome.org/projects/gup/, March 2001.

  4. Benson, C., Elman, A., Nickell, S., and Robertson, C.Z. “GNOME Human Interface Guidelines v1.0”, http://developer.gnome.org/projects/gup/hig/1.0/, August 2002.

  5. Smith, S. et al., “GNOME Baseline Usability Study :: Participant Script”, http://developer.gnome.org/projects/gup/ut1_script.html, March 2001.

  6. Sun Microsystems, Fleck, J. and Kirillov, A., “GNOME Documentation Style Guide v1.0”, http://developer.gnome.org/documents/style-guide/, 2003.

  7. Larsson, A., “Icon Theme Specification”, http://www.freedesktop.org/standards/icon-theme-spec.html, December 2002.

  8. Sun Microsystems, GNOME 2.2 User Guide, http://www.gnome.org/learn/users-guide/2.2/, 2003.

  9. Kamiyo N., Yasuhiro Y., Yoshiyuki N., Kouichi K., and Yunwen Y., "Evolution Patterns of Open-Source Software Systems and Communities," Proceedings of the Workshop on Principles of Software Evolution, International Conference on Software Engineering pp. 76-85, ACM Press, 2002.

  10. ”Gnome.org Mailing Lists”, http://mail.gnome.org/mailman/listinfo

  11. LWN.net, “The Open-HCI Project Launches”, http://lwn.net/Articles/21925/, February 2003.

  12. Sun Microsystems, “Java Look and Feel Design Guidelines; Advanced Topics”, Addison-Wesley, December 2001.

  13. Pedersen, N., “High Level GNOME Desktop User Object Model”, http://www242.pair.com/nilsp/nils/G2-desktop/desktop_user_object_model.png, December 2001.

  14. Cooper, A. and Saffo, P., “The Inmates are Running the Asylum”, Sams, April 1999.

  15. Nichols, D. and Twidale, M., “Usability and Open Source Software”, http://www.cs.waikato.ac.nz/~daven/docs/oss-wp.html, October 2002.

  16. Frishberg, N. et al., “Getting to Know You: Open Source Development Meets Usability” (CHI Workshop), http://www.acm.org/sigchi/chi2002/workshop-monday.html#six, April 2002.

  17. Pennington, H., “GNOME Personas”, http://mail.gnome.org/archives/desktop-devel-list/2002-December/msg00482.html, December 2002.

  18. McDaniel, S., “What’s Your Idea of a Mental Model”, http://www.boxesandarrows.com/archives/whats_your_idea_of_a_mental_model.php, February 2003.

  19. Preece, J. et al., “Human Computer Interaction” (Chapter 6), Addison-Wesley, 1994.

  20. Spool, J., “Tabbed Dialogs: Usability Information”, comp.human-factors newsgroup, 30th August 1996.
    (http://groups.google.com/groups?th=c7ba84de7cdcfbd8&seekm=DwxMKM.ACI%40world.std.com#s)








1Capturing users' mental models so that the conceptual model matches it as closely as possible is not always a trivial task, but it is another example of why user involvement at each stage of the design process is an important usability consideration. Mental models are an extensive topic in themselves; see [18] and [19] for examples of further reading.

2I'm not making this stuff up! This is a real example that was faced by ex-Apple UI designer Bruce Tognazzini, who now works for the Neilsen Norman Group, when he was designing a system to be installed in a busy hospital department.