November 25, 2004
How to file a good bug report
In this article I offer a few tips on how to file effective bug reports using the Wincent bug tracking database. If you take the time to file a well-written bug report then you'll be helping to get the problem fixed as quickly as possible. It's important that you follow these guidelines because incomplete or duplicate bug reports can be counter-productive and a waste of time for both the person that writes them and those that have to read them; I appreciate your efforts in filing quality reports.
There are a number of tutorials and guides out there on how to write good bug reports (for example, here and here). In this article I merely want to highlight some key points and offer some specific guidelines that apply to reporting bugs for Mac OS X software.
Search before reporting
Searching the database is very easy. On the front page of the database (bugs.wincent.com) is a search field that's labeled "Enter a bug # or some search terms". There is also a much more sophisticated search form as well as an alternative search interface, but the truth is that the default search field is usually enough to find what you're looking for.
Only file a new bug report if you cannot find an existing report that covers the same issue. The whole idea of the public database is that bugs will get squished more quickly when people share information. By comparing similarities and differences between the experiences of different users, common factors can be identified, and causes targeted.
If you file a duplicate bug report it will be closed. The best way to draw attention to an issue is to add your comments to an existing bug; in fact you can even vote for bugs that you want prioritized. I receive copies via email of all new bug reports, as well as all comments added to existing bugs. For this reason using Bugzilla is the preferred form of reporting bugs, rather than emailing me. (With Bugzilla, I am going to get the email anyway, but the bug hunting effort becomes a collaborative and therefore more effective one.)
Perform basic troubleshooting prior to filing the report
There are some steps you should take to identify the causes that trigger the bug before filing a report. For example, if the software is malfunctioning because you have a corrupt preferences file, then you'll save yourself a lot of trouble by doing some investigation beforehand and you won't even have to file a bug report. You should also consult the FAQ (Frequently Asked Questions) before filing a bug report, just in case the issue has a solution that you've overlooked (each product has its own FAQ section on the website).
- Rule out preference file corruption by shutting down the program, dragging the preferences file to the Desktop, and then restarting the program. Preference files are stored in "~/Library/Preferences/", and have names like "org.wincent.Synergy.plist" and "com.wincent.WinSwitch.plist". If the problem goes away, you know you had a corrupt preferences file. (Note: Your preferences file may contain your registration information so you should make sure that you have a copy of your license details before disposing of it entirely.)
- Try an uninstall/reinstall cycle. Sometimes in the day-to-day use of a system files get moved, deleted, and overwritten. You should try uninstalling the software (very easy if the software comes with an installer; just run the installer and click the "Uninstall" button) and then reinstalling it. If the problem goes away, you know that your previous installation was somehow damaged.
- Eliminate potential conflicts with third-party software. A good example of where third-party software can often cause problems is with Synergy, which talks to iTunes. Before reporting a bug in Synergy, you should review your system and see if you have any other third-party software installed that talks to iTunes and could be interferring. Try to reproduce the issue when the other software is uninstalled or disabled. If you can't reproduce it, then you know you've discovered a conflict.
- Try from another user account. If you don't have another user account on your machine, create one. If you can't reproduce the problem then you know something specific to your user account is triggering the problem, which narrows the scope of investigation.
- Try on another machine. If you have access to another machine try to reproduce the problem on that machine. This can help to identify factors which contribute to the problem (such as operating system versions on which the problem exists/don't exist; hardware and peripherals which appear related to the problem, and so forth).
- Try rebooting. Rebooting returns your machine to what is hopefully a "known good" state, something that may not be true if your machine has been running for hours, days or weeks. If the problem goes away then you know that something in the environment was triggering the issue.
- Check for a corrupt disk. If you are seeing generalized problems it may be a problem with a corrupt disk rather than an issue with a specific piece of software. To find out if this is the case try booting from the Mac OS X install disc (hold the "C" key while restarting with the disc in the drive) and choose "Disk Utility" from the Utilities menu; you can then click on the "First Aid" tab and try verifying and repairing your disk(s).
Include Crash Reporter logs
If your bug involves a program crash, look for a crash log in ~/Library/Logs/CrashReporter/. You don't need to include information about crashes that occurred last year or month; just look at the most recent crash. If the log entry is relatively short, you can paste it directly into your bug report. If the log is very long, you can add it to the bug report as an attachment.
Use the Activity Monitor
If the program is hung (stuck, or "beachballing") but hasn't crashed, you may be able to collect useful information by doing a sample with the Activity Monitor application (in /Application/Utilities/). Find the stuck process in the list, double-click it to get more information, and then click the "Sample" button. The output from the sample could prove useful in finding exactly where the program is stuck. If the sample output is very long, you should add it to the bug report as an attachment rather than pasting it in directly.
Include relevant system.log and console.log output
Use the Console application (in "/Applications/Utilities/") to see if anything relevant was printed to the system.log or console.log files at the time of the problem. Once again, if the amount of material is small you can paste it directly into the bug report, otherwise add it as a separate attachment.
One way to easily catch all console output is to run the application from the Terminal instead of double-clicking it in the Finder. As an example, let's imagine that you want to run a copy of Synergy Advance installed in your /Applications/ folder; you would enter the following commands (split across multiple lines for readability):
cd "/Applications/Synergy Advance.app" cd Contents/MacOS "./Synergy Advance"
Another way of doing this is to type cd followed by a space, then drag-and-drop the application icon onto the Terminal window and press the Enter key. This will change into the directory that corresponds to the application's bundle. You then could just type:
Yet another way is to right-click or control-click on the application bundle in the Finder, choose "Show package contents" from the contextual menu, navigate to the "Contents/MacOS/" folder, and then drag the application's executable file (the file inside "Contents/MacOS/") into the Terminal and press the Enter key.
Attach System Profiler reports
I do all my development using hardware and software that is as stock-standard as possible. I minimize the number of third-party products installed on the computer and I do my best not to connect any exotic hardware or peripherals. I do this because I want to know that my software is going to work on the from-the-factory hardware that 90 or 95% of Mac users have on their desktops.
By attaching a System Profiler report you can help me to identify differences between my stock-standard machine and yours, differences which may help to troubleshoot the problem.
To produce a report, open the System Profiler (in "/Applications/Utilities/"; alternatively, you can choose "About this Mac" from the Apple menu and then click the button for more information). In System Profiler, choose "Export" (as plain text) from the "File" menu. These reports are often quite large in size, so you should generally attach them to the bug rather than pasting them directly into the bug. In addition, the reports can contain large and irrelevant chunks of logs. It's generally fine to open the report in a text editor and cut out all the logs. The parts of interest are the hardware configuration and the installed software.
Even if you don't attach a report, at the very least your bug report should include information about which version of Mac OS X you are running, on what hardware, and whether you have any peripherals connected or third-party software that might conflict.
Distinguish between bug reports, crashes and feature requests
The bugzilla database is used not only for bug reports but also for feature requests. If you are making a feature request, make sure the summary begins with "REQUEST: ", so as to distinguish it from the bug reports. If you are reporting a crashing bug, start the summary with "CRASH: ", so as to make it stand out from non-crashing bugs.
Reproducibility is paramount
If you've read the bug writing guides I mentioned above like this one, then you'll know that the reproducibility is the most important factor in getting a bug squished. In contrast, the enemies of getting bugs fixed are vagueness and imprecision. Here's an example of the latter:
The app stuffs up half the time whenever I try to skip tracks. I can't remember this happening with the last version.
It's very unlikely that such a bug will be fixed because it's not reproducable. Furthermore, the description is vague and imprecise, because the reporter doesn't define what is meant by "stuff up". Does it mean that the software crashes? Freezes? Does nothing? Does something, but not what was expected? And here is the same issue, written well:
Since upgrading from 1.3 I get the spinning beachball cursor whenever I try to skip tracks using the mouse. This happens 100% of the time. After a minute or two the beachball goes away. The problem doesn't occur at all when I use the hot keys to skip tracks.
This reporter will get bonus marks if he or she also includes some background information like this:
This is on Mac OS X 10.3.4, iMac G5, no third-party hardware or peripherals connected. I am running iPod2iTunes as well. Have tried reinstalling, rebooting, and running from other user accounts, but it keeps happening.
One issue per bug report
And one final pointer: make sure that your bug reports only contain one issue per report. Multiple issues in the same report make the process of zeroing in on the causes more difficult, and they hinder collaboration. If you file a report with multiple issues you'll be asked to file separate reports for each issue.
Thanks for taking the time to read this guide, and thanks for contributing to the effort to squish those bugs!
Posted by wincent at November 25, 2004 7:26 PM