Have a Bountiful Bug Bash! Handy Tips for QA Teams

Now, with around 100 completed bug bashes under belt here at Netguru, we have some tips that will hopefully help your bug-bashers come back from the hunt with bugs aplenty.

How to Bug Bash Like a Pro – Netguru QA flow

Invite the right people...

Bug bashes should represent a diverse crowd of users, so pretty much anybody will be a good addition to your team. Try to get your group as big and varied as possible! However, there are some kinds of people who really shine during bug bashes, or who can really benefit from joining them:

• Juniors and interns of any kind – it doesn't matter if they're working in QA or in software development. During a bug bash, they’ll get a great opportunity to learn about the apps they're working on. In return, they can provide an outsider's perspective, as they may look for bugs in ways long-time employees might never think about.
• New additions to the team – it's good when new developers or managers have an opportunity to click through the app. At a bug bash, they’ll get a chance to click on everything. As an added benefit - they’ll find bugs for you!
• Quality Assurance specialists – as professional bug exterminators, they can find flaws in anything in seconds! Their experience is a great asset. Every QA specialist has a somewhat different approach, so adding one or more to your bunch can geometrically increase the number of bugs you catch. A good QA expert may be the workhorse of your bug bash, filling one bug report after another.

...and at least one Quality Assurance Specialist (QA) from the outside

The idea is that you want to have as many guys and gals as possible during a bug bash since everyone adds their own approach and strange new perspectives, increasing your chances of discovering bugs. The problem is that people have their own jobs, and if you're working in small teams it would be unfair to ask developers from other projects to stop their work help you bash “your” bugs.

Netguru has developed the following flow:

• Don't start the bug bash unless you have at least one additional QA specialist on board. We need a person from outside of the project who’s able to take a fresh look at the app. QA folks have their own work to do, but joining bug bashes in other projects is one of their responsibilities - and an opportunity to learn and to escape daily routine.
• Set the bug bash date and time together with the team in advance to ensure that everyone will have time to come, and an hour or more of bug-hunting won't stop anyone from completing their more urgent tasks.

Set the right time

Bug bashes are fun to organize, and beneficial for everyone – but some times are better for holding them than others. The best time for a bug bash is:

• when there are many of the right people around (see points 1 and 2)
• just after an important update in the app (like frontend redesigns or brand new features)
• when enough time has passed since the last bug bash (so that your team has repaired most of the bugs found last time!).

Be an over-confident oddball

On a day-to-day basis, you want to stay rational at work. When testing, you focus on likely user paths. You choose high coverage over depth. You might not have time to delve deep into things that look strange, but innocuous enough. I tend to note them down, though – notes can come in handy during a bug bash, which is a regular part of our testing flow.

During a bug bash - you do the opposite. Keep your eyes peeled for anything looking off. When you find something suspicious, strike against it with all your might! Exploit every weakness you see. Double-check the strange things you previously took note of. Try to hack into the app, even if you know almost nothing about hacking. Do stupid things: copy sources of whole websites into post titles; give yourself admin powers, then ban yourself; nothing is too ridiculous!

This is just my approach - there’s nothing preventing you from acting rationally. But since the goal of the bug bash is to try out as many user perspectives as possible, I believe that a small degree of reckless bravado is appropriate.

Get prepared

Most bug bashes don’t take long - ours typically last 1-2 hours. To ensure that as much time as possible is spent on the actual bug bashing, you can try the following:

• before the bug bash starts, make sure that the app is functional - you don’t want to invite five to ten people only to show them the homepage throwing a 500 error.
• Prepare the docs – we use shared spreadsheets with all the information necessary for bug bashers: URLs, passwords, logins, links to files with info about the app. Most importantly, the spreadsheets should contain columns for describing the bugs and for linking screencasts and screenshots (never forget these!). A spreadsheet like this works wonders for making the bug bash more efficient.
• Ask your developers to prepare, too! It’s good to have error tracking software (in our case, Rollbar) at hand, and a running console for the staging server where the bug bash is taking place. Whenever errors appear, a well-equipped dev will be able to see what’s happening and debug on short notice.
• Introduce new people to the app - show or tell them what the app is about before the bug bash starts. If you have a short, easy-to-read overview of the app in your documentation, share it with bug bash participants (this should be good motivation for you to ensure that the documentation is up-to-date).
• At the same time, don’t over-prepare the participants! First, don’t expect them to read more than 10 pages of documentation in their free time. Second, while it’s good that they know what the project is about, it’s bad to feed them the habits you developed while testing the app. The QA working on the project every day may be used to the idiosyncratic, unintuitive or illogical solutions implemented in the app, but fresh-minded bug bashers may get tripped up by them - and that’s a good thing – more UX bugs reported for you! Ultimately, you should aim for the participants to have a decent understanding of what the app should do, but have them find out how to do it on their own.

Get the maximum coverage

With many people using the app at once, bug bashes are a great opportunity for testing as many ways of interacting with the app as possible. You can assign a specific browser, device, account type, etc. to each bug-basher before the event starts and write what’s assigned to whom in the bug bash docs. It dramatically increases coverage, and makes it easier to debug and record the bugs later on.

For added effect, you can test more unusual, but still used, equipment – maybe a smartphone model with a unique screen resolution to mess up mobile apps and websites? Or else, you can try old versions of Internet Explorer (possibly with the help of Browserstack) – notorious for their poor handling of, well, everything.

Always keep in mind how the target users tend to interact with the app. If it’s a mobile project, it should have compatible devices listed. If it’s a website, what browsers are users most likely to choose? Are they high-tech specialists with top-tier equipment at hand, or do they have little knowledge of IT and/or depend on outdated devices? Likely, the owner of the app has all the info you need about the target users’ software and hardware. And even without this info, sites like Wikipedia and w3schools.com are a good start.

Get some cookies

If nobody wants to sign up for a bug bash, and your desperate pleas for help are answered with tumbleweed GIFs – offer some cookies for the bashers. It works!

What are your techniques for running a bug bash? Share them with us in the comments below!