Virtually anyone who works professionally with software (IT professionals, software developers, support analysts, quality engineers, and their managers) will eventually, if not frequently, have to submit a defect or software problem report. When software acts unexpectedly, you have to determine whether the problem is your own or someone else’s, if the problem is environmental or a user error, or if it is even a problem at all.
This ability to winnow out the root of a problem and report it accurately is a rare and valuable skill. Making sure that the cause is treated and not the symptom will save time, effort, and money.
1 – Be sure it’s a bug
Is it functioning as designed?
Software development is a relatively young and amorphous endeavor. As such, there are few standards to proscribe the way any given program should act. Developers build what they understand to be the correct solution to a particular problem, but what they build isn’t always what users of the program expect.
When users attempt to perform an operation, they expect a certain outcome. If the program doesn’t produce the outcome, the person using it might naturally presume the program has a bug. While this is sometimes the case, the program may be doing what the developer designed it to do. In this case, the program is said to be ‘functioning as designed‘ (FAD).
If you file a defect or software problem report (SPR) on a program that is FAD, you may receive a note from the developer describing the intricacies of the particular operation. Hopefully, you’re working with an articulate developer who has patience and a good sense of humor. If so, the information you get back will be enlightening and useful. If your developer friend is having a bad day/week/year, you may get a less-than-erudite explanation as to why your SPR is not worth the paper it’s written on.
In either case, it’s best to avoid writing SPRs on features that are FAD. To that end, be as certain as you can be that you understand the functionality in question. Read the documentation, talk to other users, and use tech support to make sure that the program is designed to do what you’re expecting. You should even talk to the developer if you have access. A quick discussion can save a lot of emails, memos, and erroneous SPR filings.
If you file an SPR that comes back as FAD, you might want to have the documentation team make note of the behavior somewhere in the user manual. After all, if you as a seasoned software professional thought it was a problem, the average user will probably run into it as well.
The difference between enhancements and bugs
Enhancement requests are similar to bugs in that they are requests for a program to function differently than the way it currently does. If you thought you found a bug, but the program is actually functioning as designed, you may want to file an enhancement request. The process for doing this varies between companies, but these requests often go through the marketing department. One caveat, however, is that bugs usually get fixed before enhancement requests, so if you need a fix/feature as soon as possible, you should try to pursue it as a bug.
Is the problem environmental?
All software is sensitive to the environment in which it operates, and is tested and certified in a specific configuration. The problem you’re seeing may be the result of using an unsupported operating system or web browser. Other sources of environmental issues include duplicate copies/incorrect versions of/or completely missing runtime libraries (DLL’s in Windows), erroneous or missing environment variables, inadequate user or program security access, the system not meeting minimum requirements, network connectivity issues (not being able to access a network resource), incorrect versions of the JVM or JDK, or other problems.
An easy way to isolate a problem from its environment is to try to reproduce the error on a machine that exactly meets the program’s requirements. If you don’t see the problem when running in the correct environment….bingo! By all means, avoid filing software problem reports until you are sure the environment is not the cause.
Is it user error?
We humans are all fallible. And when it comes to using certain software, we’d have to be superhuman to never make any mistakes. But there’s nothing worse in the life a software professional than having an SPR rejected as ‘user error’. Some developers take a measure of glee in marking SPR’s this way. Maybe they’re happy not to have a real problem to fix, or maybe their motivation is more malevolent.
But before you file an SPR, be absolutely certain that your expectations about how the program should perform are in-line with the program’s intent. Otherwise, your SPR’s risk the possibility of receiving the dreaded ‘user error’ status. And if you get too many of these, developers will not take you seriously, and your SPR’s – even the valid ones – may wind up being ignored.
2 – Pinpointing the problem
An accurate description
I once worked with a QA manager who threatened termination if the words ‘doesn’t work’ ever appeared in an SPR. Vagaries like ‘doesn’t look right’, ‘stopped working’, and ‘won’t run’ don’t actually describe a problem. Rather, they usually just frustrate someone who is responsible for fixing the SPR or assigning it to the right team.
A better approach is to clearly define the exact nature of the problem. If a field in a screen was blank that should have contained a calculated value, you might be tempted to say something like ‘Bonus calculation didn’t work.’ While the statement may be true, it doesn’t describe the real cause and effect. Rather, provide a detailed account of what was supposed to happen and what happened instead. Include the events leading up to the error, the action that triggered it, the expected result and the actual result, any error text the program displayed, anything that changed in your environment shortly before the problem occurred, and any other details that will help the developer pinpoint the problem.
A better description of the calculation problem might be:
‘After filling in all required fields of the Employee Bonus screen (date hired 1-13-2013, current salary 45,000 and bonus rating 4), I selected the Calculate Bonus option from the Actions menu. The hourglass spun very briefly, less than one second, but the Bonus field remained blank. No error was displayed and the program behaved as if nothing was wrong. This problem began after I updated to version 2.3 of the Salary module in our finance application. It appears when run from both an Android phone (version 4.2) and the Chrome browser. Additionally, it only appears to happen if the employee is a manager.’
This description gives the developer all he or she needs to understand the nature of the problem.
The problem scenario
Succinctly describing the process and circumstances that produced the problem is as important as identifying the problem itself. Simply put, you should include all relevant details and omit all irrelevant details. This isn’t always easy, and winnowing the problem down to its least common denominators is a skill in itself. Some questions* to answer that should help you define the problem domain:
- Is the problem reproducible at-will?
- Does the problem occur with both the current and previous versions of the software?
- If the software runs on multiple operating systems or browser, does the problem appear in every environment or only certain ones?
- Does the program interact with third party software? If so, can you ensure that the other software isn’t at fault?
- Has any software or hardware recently been installed on or removed from the system? If so, does the problem still exist if the system is restored to its previous state?
- If the program runs across the network, does the problem occur when it is run locally on a single machine?
*Note that these questions assume that the program is running in a supported environment. That is, the required operating system version and patches, third party software versions, minimum system requirements, etc., are in place.
You may not be able to answer every one of these questions, but the more relevant information you supply, the better the developer will understand the scope of the problem. Also, if the problem is reproducible, include a list of the steps required to recreate it. This will help the developer determine the cause and when the time comes, it will help whoever has to determine whether or not the problem has been fixed.
Assigning an SPR
Assigning a bug to the proper developer/team for fixing is critical in getting an issue resolved quickly. If the SPR is assigned to the wrong team, it could sit unnoticed or ignored indefinitely. Additionally, loading up a developer’s queue with problems that aren’t their responsibility will almost certainly get you, at the very least, a not-so-nice email asking you to do your homework before assigning him any more defects. If you have to assign a defect to an individual or team, do your best to pick the right one. If you don’t know, ask around. The developers will thank you.
Don’t overlook intermittent problems. Running into a problem that isn’t readily reproducible can be extremely frustrating. Our inclination is to write these bugs off as our own mistakes or some other anomaly. Often times, however, they are valid and important to fix. And because they are elusive by nature, keeping track of them becomes extra important. If you document one of these bugs in the form of a problem report, even if it doesn’t get fixed, the next person to run into it will see your SPR and should be able to build on it.
Before filing the report, be sure you’ve followed the proper procedures for installing the software, including the environment. And, as always, include all pertinent details. Also, make note of the fact that the problem is intermittent and not reproducible at-will. The developers may ask for more details and may want you to do more exploration. Keep track of all your findings in the problem report. If the bug remains unfixed for a while, you’ll save someone a lot of time recreating the conditions when the time comes to fix it.
Verifying the fix
Once you’ve identified a problem, it’s important that you articulate the means by which you will verify its solution. If you omit this step, you or the person testing the fix could have a hard time reaching a conclusion.
With some bugs, the verification will be obvious. In others, it will be require a complex and detailed series of steps that will confirm or refute that the problem has been solved. If you followed the directions outlined above when you filed the SPR, the groundwork should be laid for the verification work with the steps to reproduce. These steps are crucial to verifying that the developer has indeed fixed the bug. To verify the fix, simply follow the steps and see if the problem still occurs. If the steps to reproduce were omitted when the SPR was written, or they don’t have the appropriate level of detail, you’ll have to determine exactly how to test the fix and what evidence you’ll use to say whether or not the problem is solved.
For example, take the hypothetical bug that we described earlier. When you get a new version of the program, you’d want to test that the bug has been corrected. You might be tempted to say that if the bonus field calculates correctly, when you select the action from the menu, the issue is solved. But there’s a problem with the word ‘correctly’. If the bonus field calculates at all, the original SPR has, in fact, been fixed. If the field calculates an incorrect amount, then the original fix uncovered a new problem, as often happens, which warrants its own SPR. By clearly and accurately defining the verification criteria, you’ll insure that your SPRs cover a specific problem and only that problem.
Problem reporting is a key component of the software development lifecycle. Accurate defect descriptions, complete scope definitions, correct assignment, and appropriate verification all contribute to timely and complete problem resolutions. That means bugs are fixed quicker, so a better version of your software gets in users’ hands faster.
Joe Pescatello developed applications for PixelMEDIA from 2011-2014.