Bug Reports

[Hare]

What are good methods with which to handle bug reports, and what is a good rate of fixing bugs? 

We had about 300 bugs reports resolved in the past 2 months, and average of 5 resolves per day. There are usually about 30-50 open reports at any given time.
The reason there are so many bug reports is because of the rate of development (we release a lot of new features).
We could reduce the number of reports in the 2 month period by spacing out new things over a greater period of time. That would result in fewer bug reports open at any given time. 

We're also trying to come up with any improvements we can make to our reporting system. 
We have a regular bug reports forum with open and resolved topics. A staff or the player who created the topic can open or resolve it if needed (like if a player finds that their bug was fixed without staff help, they can resolve it on their own, if we marked it resolved but it comes up again, they can re-open it). Higher priority bugs are pinned. 
Then we had a modbox for sensitive things. 

Sometimes we have bugs that we have trouble figuring out. I hate to say it, but we do have some bugs that have been going on for a month or more and we've yet to figure out how to fix it. What's best to do in these situations? In the past, we've managed to 'finally' fix something after a few months, or recoded the feature around it in a way that removed the bug. 

Other random questions:
- When do you shut down a feature?
- When do you reimburse a player for lost stuff due to a bug?
- Do you temporarily take and hold things that a player reports as a potential bug until it can be determined whether it was a bug or not?
- How do you prioritize bugs?
- How do you communicate with players before, during, and after the bug is fixed?
- How long is reasonable for players to wait on various types of bugs?
- What is done about a bug that removes things from a player's account (like money) that they say they need right away?
- Any other comments or ideas on how to have a good bug report system?

Edited October 8, 2018 by Hare

[Matthew]

1. Do you have a test suite implemented for your code base? I find that automated tests before deploying a new version of the game catch most bugs which will help reduce the number of bug reports you have to deal with. As for what a good rate of fixing bugs is, that depends entirely on your team and playerbase. I'd say fix the bugs that are going to have the most impact on your revenue first.

2. I don't typically shut down a feature as such, if a test fails I simply don't deploy the code until it's fixed, but if it somehow breaks in production I immediately revert the change to get back to a known stable state.

3. If I can reproduce the bug and confirm it's had an impact on the player, they get reimbursed with no further questions.

4. No, I've never temporarily taken and held something to determine whether it's a bug or not.

5. By the impact on revenue. If a bug is going to cause me to lose revenue, I need it fixed ASAP.

6. I add them to an internal issue tracker where they can get updates by email.

7. Depends on the type of bug. If it's a cosmetic issue, it's probably not a high priority and they can wait a few weeks. If it's a systems issue (e.g. causing items to disappear) or a payments issue (e.g. they get charged twice or something), then it's a high priority and I aim to get it fixed within hours.

8. I give them the thing back, make a note, and ask them to avoid using that system until I can deploy a patch.

9. The best bug report system are automated tests run before you deploy new code to your servers. :-) They'll help catch regressions, etc. before your players do.

[Dinocanid]

• - I usually test features privately before having them live on the site, usually through local testing like XAMPP, although it can't be helped that bugs arise when making a feature public. I've never had to do it, but I'd only shut down a feature if it's causing a lot of users grief and/or it's ruining other aspects of the site. During the downtime I would take user input into account and re-release the feature when it is better.
• - Pretty much what matthew said. If I can reproduce the bug then I'll reimburse any affected users ASAP. If the bug can't be proven or reproduced by anyone though, then they don't. (People can and will lie about a valuable pet or item disappearing in order to obtain it for free)
• - I've never done this. I would only do this if a user was able to obtain things through illicit means (like scams, hacking, or exploiting of bugs)
• - I prioritize them from game-breaking to minor inconvenience. If a bug causes major problems I would prioritize that over, say, a bug that makes pets appear upside-down.
  • That being said, if a bug is extremely minor and easy to fix, I'll actually prioritize that over a larger bug since I can fix it in about 10mins and go about fixing the other ones afterwards. 
• - My method goes like this:
  • Respond to let them know I've acknowledged the bug exists (and possibly the approximate cause if I know it)
  • Give regular updates as I'm in the process of fixing a bug (like if it's going well, if I'm still figuring out the source, etc.)
  • Mark the topic or help ticket with a "resolves" tag once the bug has been fixed.
• - It depends honestly. Development is pretty slow for me, so I usually jump on bugs as soon as someone reports it and it's done within a day or so. I would the wait time should be relevant to how complex the bug is. If it's just a spelling error or something, it shouldn't take a month to fix.
• - I would prioritize a bug like this pretty high and look into it as soon as I'm able. If it's confirmed to be a real bug, I'll add the money back to their account and inform them to avoid the bug until a fix can be made.
• - I plan on having built-in help ticket systems on my site so I can easily track and label bugs (preferably a setup similar to github's bug section), but that's it. I would say test features extensively before adding them to the site as well (preferably with a team dedicated to testing new features, since they're more likely to spot bugs than one person alone), but of course it can be hard to predict what one out of thousands of people will do so some bugs are only spotted after release. Having an easy to navigate and use place for bug reporting will help keep things organized and under control.

Edited October 11, 2018 by Dinocanid

[Hare]

Thank you for your responses! Very helpful.

We don't have any automated tests that I know of (what are these exactly?). We check error logs and test out the features in various different ways before going live. 99% of our bugs are similar to this one:

For example, our most recent one:  angora rabbits have to be groomed to show, and their groom meter starts to tic once they're born. This is set up so they'll need grooming by the time they're 6 months old (adults). We have a 'low maintenance coat' you can breed for though that makes it take longer for them to get matted. This result in some 3-4 month old angoras being showable (which was reported as bug because in real life they generally don't have a full coat by then).
We didn't code the shows to just exclude the younger angoras at that age because it would have required a ton of exceptions to list all the angora breeds (and then we're worried we'd forget to update it in the future when we add more angora breeds). So I fixed the bug by not applying the low maintenance coat to animals under 6 months. 

Would an automated tester be able to help with this? Having a dedicated testing group sounds like a beter way of handling new features than releasing them site-wide, we should implement that and I'm thinking have a longer testing period after coding new things.

[Matthew]

2 minutes ago, Hare said:

We don't have any automated tests that I know of (what are these exactly?).

Would an automated tester be able to help with this? Having a dedicated testing group sounds like a beter way of handling new features than releasing them site-wide, we should implement that and I'm thinking have a longer testing period after coding new things.

Automated tests are bits of code that you write that you then run before uploading the codebase to your site. In other words, you write the tests in the style of how you expect the code to be, and the tests confirm whether that is actually true or not. For example, below is a piece of code I wrote to check that my emails are being sent out with the correct details. The language is different but the principle is the same.

describe "new_forum_post" do
    let(:mail) { ForumPostNotification.new_forum_post(forum_post(:first), users(:matthew).email) }

    it "renders the headers" do
      expect(mail.subject).to eq("A new forum post was made by example!")
      expect(mail.to).to eq(["[email protected]"])
      expect(mail.from).to eq(["[email protected]"])
    end
  
    it "renders the body" do
      expect(mail.body).to include("A new forum post was made!")

    it "includes the unsubscribe link" do
      expect(mail.body).to include("https://example.com/notifications/unsubscribe")
    end
  end

These tests run every time I make a change in the codebase, as well as on upload to GitHub (code storage) and the servers. If any of the tests fail (e.g. perhaps the email no longer includes the unsubscribe link, or maybe the subject has changed), the upload will stop and I need to fix whatever is broken before I can upload again.

These save you a lot of time and effort over manual testers. Manual testers are important, but it's not realistic to expect people to go and test out every part of the site to make sure nothing has ever broken. These automated tests will probably cut down on the number of regressions (where a feature stops working) and bugs you encounter.

Not sure what language you are using, but there are likely to be things that can do this in PHP for example.

[Hare]

3 hours ago, Matthew said:

Automated tests are bits of code that you write that you then run before uploading the codebase to your site. In other words, you write the tests in the style of how you expect the code to be, and the tests confirm whether that is actually true or not. For example, below is a piece of code I wrote to check that my emails are being sent out with the correct details. The language is different but the principle is the same.

describe "new_forum_post" do
    let(:mail) { ForumPostNotification.new_forum_post(forum_post(:first), users(:matthew).email) }

    it "renders the headers" do
      expect(mail.subject).to eq("A new forum post was made by example!")
      expect(mail.to).to eq(["[email protected]"])
      expect(mail.from).to eq(["[email protected]"])
    end
  
    it "renders the body" do
      expect(mail.body).to include("A new forum post was made!")

    it "includes the unsubscribe link" do
      expect(mail.body).to include("https://example.com/notifications/unsubscribe")
    end
  end

These tests run every time I make a change in the codebase, as well as on upload to GitHub (code storage) and the servers. If any of the tests fail (e.g. perhaps the email no longer includes the unsubscribe link, or maybe the subject has changed), the upload will stop and I need to fix whatever is broken before I can upload again.

These save you a lot of time and effort over manual testers. Manual testers are important, but it's not realistic to expect people to go and test out every part of the site to make sure nothing has ever broken. These automated tests will probably cut down on the number of regressions (where a feature stops working) and bugs you encounter.

Not sure what language you are using, but there are likely to be things that can do this in PHP for example.

Thank you! This looks about like what we have. We could add more. =D

[hurricaneviolet]

Honestly, I think the short and simple answer is that your bug reporting system is fine as long as you can easily find what people are reporting and people can figure out how to report things.

On HP we have a section in the forum which is good because it makes it all super public so other people can see the issues and comment if they have the same issue, and they can see staff responses to it. People also message me about bugs, and I keep a separate list.

As far as a good rate of bug fixing, it depends on the bug. I think in general the best time to fix a bug is ASAP, though. At least on a live game.

Quote

- When do you shut down a feature?

When it no longer serves the game. I can't think of a time when I've shut down a feature because it was too bugged (although, I can't say for sure that it's NEVER happened - 9 years is a long time to remember things :P), but we have completely revamped and changed features that were chronically bugged.

Quote

- When do you reimburse a player for lost stuff due to a bug?

ALWAYS.

Quote

- Do you temporarily take and hold things that a player reports as a potential bug until it can be determined whether it was a bug or not?

I'm not sure what you mean here. If you mean if a player reports something like "I think the bank gave me too much money, is this a bug?" no, I don't hold those funds. I either investigate and reverse it once I find out that it was a bug, or if it was wide-spread enough and small enough, I let it go because it was our fault. We once had a bug where anytime someone used an upgrade coupon it was giving them 25 mill in game money on top of it. I was pretty sure based on what the bug ended up being that it had always done that, but I only knew WITHOUT A DOUBT that it has happened for the last two coupons, so I only reserved it for the last two coupons.

Quote

- How do you prioritize bugs?

If it's affecting gameplay in a major way then it is an ASAP type of thing. If it's something I can fix easily, I do it as soon as I have time. If it's something I need to hire someone to fix and it's not affecting gameplay, but are sort of annoying, I deal with them the next time we have other bug fixes or programming done.

Quote

- How do you communicate with players before, during, and after the bug is fixed?

In the news, on social media, and on the forums if it's widespread, or I just message the reporting player and let them know if it only affected them.

Quote

- How long is reasonable for players to wait on various types of bugs?

It really just depends on the bug and how it's affecting them. If they can't play major parts of the game, any amount of time beyond you finding out and contacting a programmer (or looking at it yourself) is unreasonable to them. If it's a small bug (for example when they buy feed it tells them "Congrats, you bought a cat!" but still gives them feed), then there's not really an urgency, but you should still deal with it as soon as you can.

Quote

- What is done about a bug that removes things from a player's account (like money) that they say they need right away?

You should start compensating players as soon as you can. If you're not able to compensate players right away, you should be keeping them up-to-date in the news (or however you communicate with your players) about the fix and when they can expect compensation.

Quote

- Any other comments or ideas on how to have a good bug report system?

Not beyond what I have already said.

Hope that helps you!

[Hare]

Thank you for your input, hurricaneviolet. We've taken a lot of the input provided here and improved our bug fixing system. Much appreciated.