QA guide
From ReservesDirect: Open Source EReserves System
How to Contribute to QA and Testing
There are three primary ways to contributing to QA: suggesting new features, developing testing scenarios, doing the testing itself.
Suggestions for new features and updates to existing features come from the QA group, from project managers, and from developers. The QA group is a very important contributor to the new functionality list, since they are the most familiar with the end-user experience of the system. Requests for new functionality should be submitted to the Requested Features page. From there, the core group of project managers will decide on what to move into the Development Priorities page. Development institutions will pick up features or functionality to develop from this page and determine which version to target for completing the functionality.
The second task of the QA group is developing scenarios. Scenarios are important to the testing process since they provide benchmarks for functionality that simply has to work in order for the system to be viable. Whenever a new version is released, a tester should be able to run through all of the existing scenarios without encountering any bugs. In addition, new features can be incorporated into existing scenarios, or new scenarios can be created if existing scenarios do not cover the new functionality.
Whenever a new version or release is ready to test, an announcement will go out to the QA listserv with a link to the test site and a list of new features. This announcement will also include a timeframe for completing the initial phase of testing. It is important to adhere to this timeframe so that developers have enough time to fix any bugs that are discovered and have them retested before the stated release date. Once this announcement goes out, testers should begin updating current scenarios and creating any new ones, if necessary. This work will take place directly in the wiki and be coordinated either on the listserv or on the "discussion" tab of the Scenarios page.
Once the scenarios are updated, a full battery of testing should begin. Members of the QA group can split up the scenarios amongst themselves however they think it will work best. This initial round of testing can involve communication over the QA listserv to see if potential bugs are really bugs and see if others can confirm them. Ultimately, actual bugs (meaning the scenarios break down at some point, for whatever reason) should be reported to the Mantis bug tracker for the RD project.
When reporting bugs to Mantis, be as specific as possible about what happened and how you produced the behavior, how repeatable it is, and how severe the problem is. If it's a tweak, report it as a tweak. If the system crashes, report it as a crash. A member of the project management group will try to reproduce the bug one more time before assigning it to a developer. Remember to search through Mantis to see what bugs have been reported already. If a bug you have discovered has already been reported and you have more to say about it, just add a note to the issue.
After an issue has been resolved by a developer, the result will be confirmed a final time by a member of the project management group.
Guidelines for Testing
Here's a few pointers for testing effectively:
- Be thorough. Test the new features and functionality, but also make sure the basics work as well. Sometimes a new piece of code can break something in an entirely different area.
- Be a detective. When you discover a bug, try to reproduce it and test it in several different ways. Test related functionality as well, to see if the problem is isolated. Make connections, but be careful of drawing conclusions.
- Be creative. Try to find workarounds to problems you see. Try to distinguish between bugs and holes in functionality.
- Be specific. When you report a problem, give detailed information on exactly what you did and what happened, and what you expected to happen.
- Be accurate. If a bug seems relatively minor, don't report it as major because it is something you personally want fixed as soon as possible. By the same token, if you run into a crash or other major problem, report it as soon as possible and raise all the flags you can so that developers can jump on it and push it back out for re-testing as soon as possible.
- Be responsive. Try to respond to questions or requests for feedback in a timely way. After a developer asks for feedback, they may not touch the bug again until you respond!
