Guidelines for Running a Bug Bash

My previous blog on Bug Bashing covered the what, why and when. The focus of this blog is how to run a successful bug bash: what you will do before, during and after the Bug Bash.

General

• A Bug Bash should be short, a period of 60 minutes should be sufficient. 10 minutes for the introduction, 30 minutes for the bug bash, an additional 20 minutes should be allotted for the bug bash debrief.
• The organiser will typically be the tester from the relevant team but could be facilitated by others team members such as the Product Manager, UX or team lead.
• It is better if the organiser does not participate in the bug bash, instead the organiser supports users and answers any questions.
• A Bug Bash should have a minimum of 5 participants to ensure there is sufficiently diverse feedback.
• Decide what channels(emails, slack etc)  will be used for Bug Bash communications( invites, updates, questions) and inform participants. Slack, yammer, or communicator are good for asking questions but I have found it unsuitable for people raising issues(i.e sharing multiple screenshots and stack traces etc). Participants should be encouraged to only use these channels for questions.

Before

• Endeavour to give participants plenty of notice but make them aware that the scheduling of the bug bash can potentially change due to unforeseen issues such as environmental changes or change of release timelines. Thus participants may need to exercise a degree of flexibility.
• Endeavour to have the team members who developed the software be present to support users and observe the feedback first hand.
• Provide any relevant documentation to the participants.
• Ensure Participant should  have access to any relevant systems and   have their own account and data(avoids overlapping with other users).
• Try to involve people from outside the team and outside your engineering department, people that have not actively been involved in the development of the product. These people are seeing the product for the first time and can give valid feedback regarding functionality, usability, and performance. People who work in customer support, training and sales are often excellent contributors.
• For obvious reasons, the environment used during the Bug Bash should be available and stable. The environment may be a test environment or perhaps a production environment with a feature toggle.
• If test environments are shared with other teams, ensure these teams are aware of the Bug Bash to prevent redeployments and other related issues.
• If the product has a web front-end and needs to support multiple browser versions (or even devices) ensure you perform an inventory of what the participants have available and assign browser version(device) to the users, to ensure user use a diverse range of browsers and/or devices. The same advice applies to testing mobile apps.
• If the system or functionality is large you can assign specific functional areas to each of the participants or ask them to complete a particular user flow. Although ensure that you are not overly restricting your participants.
• In your defect management tool, perhaps create a category for bug bash defects. For example in Jira Create a bug bash story or label in Jira so there is somewhere to group potential issues. This is also useful for tracking the impact Bug Bashes are having on your bug metrics.
• Participants should be directed towards issues that would impact s customers and other key stakeholders, rather than edge cases/obscure issues. Also feedback on feature improvements should be encouraged.
• Create a document for collecting feedback and a shared folder for uploading images, videos, logs. Typically I share a simple google sheet and shared google drive folder for artefacts. The google sheet has tabs for the different functional areas  and a few columns to capture the person name, description of the issue and link to any artefacts.
• If you are finding it difficult to find volunteers, offering food (pizza, cookies) or a reward for best bug can increase participation.
• I am a huge advocate of Heuristics and Mnemonics, i will often suggest heuristics that may apply to the feature for the Bug Bash and hand out the testheuristicscheatsheetv1.pdf  to participants.

During

• Open the Bug Bash by explaining why you are running the bug bash and give a brief overview of the feature to be bug bashed. Often it make sense to demonstrate the functionality.
• If changing existing functionality, it may be worthwhile noting any potential risks.
• After the initial overview of the functionality, set a timer for 20 minutes and states this time is for focused testing. (Repeat for another 10 minutes).
• Ensure that you inform all participants of what you expect from them in case they believe they have found an issue.
  • If an issue is found during a bug bash, a participant should not spend time investigating the issue.
  • Instead simply note the problem in the feedback sheet and continue on with further testing. This maximizes testing time and avoids investigation of known issues.

• The participants of the Bug Bash can perform the work from their own desk, but in some cases, it might be good to have all participants in a single room as this has the following benefits:
  • Any issue that is encountered that affects all participants this can easily be communicated.
  • It ensures the participants are not distracted by their regular day-to-day work or by other colleagues

After

• Allow Approximately 20 minutes for debriefing so that all participants can provide feedback on any issues that have been captured.
• Gather feedback immediately after the Bug Bash is completed when the information is fresh in peoples minds.
• Rewarding defects. Typically there is a prize for “Best Bug” found during the Bug Bash. Personally, I have found that praising people who find important issues and updating them when the defects are fixed has a more motivating effect than a prize. I have heard people say “It’s great to know that I made a difference to the product”
• Metrics: ensure that the issues that were discovered in the Bug Bash are correctly labeled. This will aid support for a  future bug Bash. Triage the Bug bash defects and ensure action is taken to resolve the highest priority defects.

Summary

This blog focused on how to run a successful bug bash, what you will do before, during and after the Bug Bash.

• They key to success is in the preparation including finding a diverse range of participants, ensuring a stable environment is available and participants understand what their role is in the Bug Bash.
• During the Bug Bash keep participants focused on exploring the software rather than investigating issues encountered.
• After the Bug Bash, capture all information immediately from or participants. Get support for a future Bug Bash by resolving any issues encountered and use these metrics to gain support from your Management team.

---

This is a personal blog. The opinions expressed here represent my own and not those of my current or previous employers.