The future of bugs.slimdevices.com

[Mark Miksis]

Maybe simply enabling confirmation will solve many of the problems of which I'm afraid, then.

I'm not convinced that this would solve the employee workload problem. If bug triage doesn't happen, then it doesn't happen. I don't see any real difference between a bug that lingers as NEW and untargetted vs. a bug that lingers as UNCONFIRMED.

Even in my most 'closed' notions, there are still non-employees submitting bugs and even confirming them. It's just a choice of how many, how they are selected, and where the basic bugzilla user fits.

I would be concerned that this change will just shift the workload from QA to support since more bug reports will need to be filtered through them.

[erland]

I've always felt you have two problems with bugzilla:
1. It contains a lot of open bugs and enhancments that's never going to be implemented
2. It contains some issues that should be handled through the support organization

Regarding problem 1:
===================
I suppose one reason to keep them open is to avoid getting duplicates because people can't find an existing bug/enhancement and register a new one. However, the disadvantage is that there are a lot of open bugs in the system that make it harder to manage.
It is to make them visible when searching, but it would also be good if you gave some kind of indication that you aren't going to implement them so you know you don't have to care about them when managing bugzilla. I suppose targetMilestone=Future might be this indication even though it sounds like something else.

Regarding problem 2:
===================
It feels like it should be pretty fast for someone with experience of your products to:
- Direct bugs that feels like support issues to the normal support organization (which will raise their load)
- Request more information from reporter on bugs that aren't described in such way that they are possible to reproduce/find. If there isn't any clear description about how to reproduce a bug or there isn't any direct reference in which part of the code the problem is, the reporter hasn't registered enough information. You could just close it and ask the reporter to reopen it when he/she has supplied more information.

I don't think you need to do the actual testing to do this classification, looking at the information in the bug should be enough.

Some statistics from Bugzilla:
==============================
I just took a look in bugzilla (bugs since 1 November), I'm not sure if this period is representative but the statistics says:
- Totally 87 bugs registered
- 64 by third party contributors
- 23 by Logitech employees
- 37 if those registered by third party contributors has been assigned a target release, most of them 7.4.x or 7.5.0, which I guess means that they were real bugs not yet found by Logitech employees.

If this is representative, it means that someone will need to have a quick look at less than 10 third party contributed bugs per day to decide if:
- It's a support issue
- It needs more information to be possible to be reproduced

If it's faster to do this classification compared to finding the 37 targeted third party bugs with your own test team, it means to me that it will be better to let third party contributors keep register bugs.

Is your feeling that the Radio quality would have been better if you had closed bugzilla for third party contributions and used the bugzilla managmement time for testing instead ?
If that's your feeling, it sounds like you don't get much from third party contributors and you should probably just allow bugs from Logitech employees.

I'm pretty sure managing the third party contributions is worth the time, but that's just my personal opinion.

Requiring confirmation on all bugs means that someone at Logitech is going to spend time verifying a lot of bugs that already is clear enough to immediately hand over to a developer. Of course the confirmation step could be the same as the classification step I mentioned above, just checking that the bug contains enough information.

Also make sure that your support organization is ready for the higher load before you start directing bugs to them. If not, you will just move the problem to another place within your organization.

My suggestion:
==============
- Keep bugzilla open for everyone
- Let someone spend maximum 30-60 minutes per day to do the classification (it might even possible to get some of the experienced third party contributors to help with this if they get some sort of compensation)
- Keep targeting all registered bugs that has been classified
- It would be good if there was some way to see if a bug has been reported by a experienced third party contributor or a new user. Bugs registered by experienced third party contributors or Logitech employees doesn't have to go through the classification step, they just needs the targeting.
- Developers should only have to look at bugs that:
- Has been classified
- Has been targeted
- It shouldn't be the responsibility of the individual developer to do classification or targeting, this is the work for someone else in most matured organizations.
- Patches registered on bugs should always mean that the bug go through the targeting step again. The reason is that this encourage third party developers to post patches and thus takes some load of your own developers. A patch doesn't automatically mean that the bug should be included in the next release, because it might need a lot of testing work, but you should at least always consider it.

Regarding community involvement:
================================
You have to decide what you want to use the community for. You have to realize that the reason many of the experienced third party contributors are here is because they feel that they are part of your development process, if you close bugzilla and developers stops communicating through the forum the experienced third party contributors are going to disappear. For me personally, the communication with the developers is most important, for other contributors I think the access to bugzilla is most important. We are here to help, you just have to show that you want it. If you don't communicate with us, there is a risk some of us will interpret that as you don't want us here, with the result that we will soon be gone.

I wonder what people would think about your products if all third party plugins/applets suddenly disappeared because the third party developers no longer are interested ?

So I think you need to be careful what you decide and realize that it might affect which people continues to be part of the community. The difference between this community and many other communities is that there are a lot of experienced users here which is willing to help and can answer many questions which takes some load of your support and QA organization.

However, I've during the last year started to feel that you really don't know what you want to use the community for and you aren't sure if it's really worth the trouble. I'm very sure that adding more resources to your own organization compared to using the community for testing activities would be more effective, but I've also got a feeling that's not an option due to costs.

I think you need to do something to engage the community more if you like to keep it at its current state. Communication is critical to do this, no communication will soon result in a community with no experienced users, just a lot of new users which won't help you at all.

[kdf]

On 2009-11-10, at 8:37 PM, Mark Miksis wrote:

>
> ChrisOwens;483990 Wrote:
>> Maybe simply enabling confirmation will solve many of the problems of
>> which I'm afraid, then.
>
> I'm not convinced that this would solve the employee workload problem.
> If bug triage doesn't happen, then it doesn't happen. I don't see any
> real difference between a bug that lingers as NEW and untargetted vs. a
> bug that lingers as UNCONFIRMED.

Confirmation alone won't really do it. You would still rely on someone keeping up to date with new reports before they get lost in newer ones, or duplicates. Maybe requiring the url field to be filled in for new reports may help. Then at least you'd require a general user to open the discussion on the forum to get some idea if it's a bug or simply something other users can sort out by sharing tips.

>
>> Even in my most 'closed' notions, there are still non-employees
>> submitting bugs and even confirming them. It's just a choice of how
>> many, how they are selected, and where the basic bugzilla user fits.
>
> I would be concerned that this change will just shift the workload from
> QA to support since more bug reports will need to be filtered through
> them.

If you leave it open and still want no workload, then all you end up doing is ignoring users who are made to believe that by filing a bug they will get it addressed.

No triage and it ends up with multiples of the same issue, misleading wording, and an increasing pile of junk that no one will bother taking on as it just goes too far. Kinda what it was like before Chris came along, when I think of it. At least back then, it was only 3000 bugs or so. Someone might just about keep that many sorted in one's head...almost.

If there is triage and it's not QA or support, clearly that means the load is on the developers. The workload exists no matter how you spin it. Someone does it or not. There have been times when bugzilla was mostly in the hands of a third party, but I would be surprised if that is the preferred way now.

-kdf

[peterw]

Chris, thanks for soliciting feedback on this.

My inclination is to at least change the Bugzilla settings so that new bugs are not automatically "confirmed" and must be confirmed by an employee or trusted user OR receive a certain number of votes before the QA team pays attention to them. (I've posted about this in the past, but now I have an urgent motivation). However, there's concern that this will create a lot of unconfirmed bugs that will just annoy users by not showing any progress.

I like this approach, and hope that you could tweak the Bugzilla UI to set expectations. Let bug submitters know what it takes to get Logitech staff to look at a bug (# of votes, or having a trusted community member "confirm" the bug). Let your staff know they're under no obligation to read bugs that don't meet those criteria. Customers with important bugs should call support or raise awareness on the forums if needed to get votes or call support to get a Logi staffer to take a look.

Systems like Bugzilla can be used as self-service resources, and I think that's the case even with the SD bugzilla system. Customers often chime in, explain current constraints, post patches, suggest alternatives like using 3rd party plugins to meet a user's desire. If you make it a hassle to submit bugs (become a trusted user, go through an "application" process, etc.), you'll 1) not hear about some bugs 2) not garner some good ideas for enhancements 3) deny customers a valuable support channel and 4) start taking in bugs through more expensive channels. By expensive channels I mean the support staff. I fully expect that it would take me considerably longer to communicate many bugs to support staff than to enter them in bugzilla.

I think Erland is right on target with his community comments. If you want help from members of the community, it's important to make those people feel wanted and feel like their time is well spent.

Thanks again.

[Philip Meyer]

>maybe for the vast majority of users, the forums and tech
>support provide enough of a bug-reporting mechanism.
>
I feel that in the last 1/2 year, there has been more of an emphasis on raising bugs in bugzilla. Bugs discussed purely in the forums have perhaps not been noticed by developers; it's only when a summary of the bug has been filed in bugzilla that issues have been addressed.

I would have thought that spending a little time going through bugzilla would be more beneficial than spending time in the weeds of the forum.

But you really need to decide whether you want people to use bugzilla to:
1) File specific bugs
2) File enhancement requests
3) Report bugs without knowing what the problem is

i.e. the forums can be used to ask questions, find solutions, work out what the change should be, and then log a specific bug. Reporting a bug like "Software crashes randomly" is not really a bug report, and requires support/forum chat to progress.

[pippin]

>> Hm, not to sound negative, but should that not be _your_ issue? I mean:
>> not yours personally, but should not Logitech decide how they want to
>> involve users?

It is mine personally and Logitech's. The team here has been given direction from our management, and I'm trying to find a 'sweet spot' where we can do what they want, and yet not lose the great value that the community has brought to the company and to the products. I would argue that the Squeezebox product would not exist today without the help of the community.

OK, in this case we can forget about marketing input for a moment.

Do I get this right that your main issue is that you have to reduce workload while you want to preserve feedback?

Now there is _one_ thing I would still like to understand (and don't) and that is, how you are supposed to find errors if not using the current mechanism.
I mean, it's not like the Squeezebox right now is a particularly bug-free system.
As I see it, there are two ways: either you rely on the bugs your users, developers and beta testers find and put into bugzilla. If you cut this down, you will have less bugs in bugzilla but no fewer bugs in your product.

An alternative to this is things like automatic testing and professional, systematic tests. In the corporate world that _I_ come from, we use these a lot and do make pretty good experience with this kind of approach. You do get much better results than from pure coincidence bug finding.

BUT

In that world we do have things like requirements, use cases, test casesm specifications, interface documents and protocol specifications. All things I haven't seen (with a few exceptions, maybe) in your world, so I don't think this is an option.

So _my_ view is, unless you do some work on the development process (software, that is), which is probably not on you to decide, you have little alternative to making as much use of bugzilla as you can, so the main focus really has to be on how to get this more efficient.

[ChrisOwens]

>> I would be concerned that this change will just shift the workload from
>> QA to support since more bug reports will need to be filtered through
>> them.

Yep that's part of my direction from management, they want support to do support and QA to do QA. And I agree with them.

>> An alternative to [the current use of community testers] is
>> things like automatic testing and professional, systematic tests. In the
>> corporate world that _I_ come from, we use these a lot and do make
>> pretty good experience with this kind of approach. You do get much
>> better results than from pure coincidence bug finding.

That is, in fact, the direction we're moving. It had been my hope that we could do BOTH; that is, manage the community even BETTER, AND have more internal testing. And maybe we can get back to that in the future. However, right now, I'm resource-limited and being asked to focus on our internal, formal testing.

>> You have to decide what you want to use the community for. You have to
>> realize that the reason many of the experienced third party contributors
>> are here is because they feel that they are part of your development
>> process, if you close bugzilla and developers stops communicating
>> through the forum the experienced third party contributors are going to
>> disappear. For me personally, the communication with the developers is
>> most important, for other contributors I think the access to bugzilla is
>> most important. We are here to help, you just have to show that you want
>> it. If you don't communicate with us, there is a risk some of us will
>> interpret that as you don't want us here, with the result that we will
>> soon be gone.

I think all the employees on both the services/software side and the products side want to keep the community. Some of the managers in our respective new organizations don't see the value (which is disappointing), but we're basically planning to manage our managers. We're a willfull team. I don't represent the developers, but none of THEM have said to me that they plan to communicate less on the forums. I consider this thread to be about the relatively narrow-scope problem of Bugzilla.

Thanks again for your feedback.

[brucegrr]

I have participated in the Squeezebox forum going on 4 years. I have never filed a bug report. Voted on a few but never filed one. I have seen a number of frustrated end-users who have had problems with their hardware, server software, plug-ins assume that there is a bug and off they go and file a bug report. I can only imagine how frustrating it is on your end to try and filter out the legitimate bug reports from the pissed off user reports.

So...how do you fix that? I don't think it is wise to shut off the forum community from bugzilla BUT I don't think bugzilla should be a place for frustrated/upset/unhappy end-users to arbitrarily file bug reports. Balance is what is needed.

I would also want to know how a "trusted user" or "trusted contributor" would be defined.

I have been a computer user since 1991. I work in the tech industry and by far the Squeezebox community is the most helpful I have ever been involved with. I would not want to see any action that diminished this forum.

Bruce

We've always allowed anyone to create a bug and we've always dealt with every bug, to a greater or lesser extent.

[pippin]

That is, in fact, the direction we're moving. It had been my hope that we could do BOTH; that is, manage the community even BETTER, AND have more internal testing. And maybe we can get back to that in the future. However, right now, I'm resource-limited and being asked to focus on our internal, formal testing.

Oh, and I don't think it's necessarily a conflict. You will be so much more efficient doing systematic testing that you will probably find the time to care about the (then much less) bugs.

HOWEVER, as I stated above, IMHO you have a long way to go to be even able to consider formal testing.

If all these changes mean that you are actually going to have more of a formal development process enabling you to do this I actually believe that this is a very, very good thing and that you are on the right way.

If not, you're screwed...

[erland]

I consider this thread to be about the relatively narrow-scope problem of Bugzilla.

I'll repeat what pippin said initially "define your development process first and then define your tool."

For most of us it really doesn't matter how you decide to use bugzilla, what you really has to decide is:
- What kind of help/input do you want from the community ?
- What kind of control do you want over that input ?
- What do you do best with your internal team and which parts do you get most out of the community ?

We have no idea how you work internally, so these are questions you have to answer within your organization.

In all large projects I've worked in there have been a person or group of people that have looked at new bugs to decide:
- If it has enough information to be useful/reproducible for a developer
- If it's worth fixing and in that case in which release it needs to be fixed

The main reason for having this "filter function" is to make sure the developers are focused on fixing bugs that are important and needs to be fixed in this release. Without the filtering, the developers will focus on what's fun, interesting, easy and not necessary what's most important.