PDA

View Full Version : Slimserver software engineering



ChrisOwens
2007-02-23, 14:04
Due to some recent questions, I decided to post a new message here about a few of the details of a software engineering project. I'm thinking specifically of Slimserver, but most of this information is applicable to other projects as well, both proprietary and open source.

Some background info:

This link contains information about what's going to be in upcoming versions of Slimserver: http://wiki.slimdevices.com/index.cgi?SoftwareRoadmap. Typically these are new features. For information about what bugs will be fixed, you can use the search feature on http://bugs.slimdevices.com, which is our bug tracking system (running software called "Bugzilla").

Slim Devices uses a "revision control" system (sometimes called "source control" or the "source tree") called "Subversion" (or "svn" for short) to store the source code for Slimserver. Detailed information about revision control is available at http://en.wikipedia.org/wiki/Revision_control if you are interested. The jargon of using a revision control system will often appear in our conversations about other aspects of Slimserver.

When a developer (either an employee or a contributor) makes a change to source code stored in a revision control system, they have to "commit" (also called "checking in") their change. This is part of the system's mechanism to make sure two or more people don't check in incompatible changes to the same part of the code. In addition, revision control lets us have "branches" in the code. A branch starts off as an exact copy of all the source files needed to build Slimserver. From that point, the source files can be changed independently. So, next time you see a new branch appear in the nightlies, you'll know, generally, what the differences are between branches. Not much. It's only after divergent changes have time to start happening that noticeable differences appear.

The main reason we create branches is to try to have a more stable software product. Most development goes on in a branch called the "trunk" (or in some systems "main"). Typically, when we get close to where we want to have a release (such as the recent 6.5.1 release) we will create a "branch" from the "trunk". This way, risky development can continue in trunk without causing major bugs in the branch about to be released.

Since perl is not a language that needs to be compiled, it's possible for developers (and brave users) to just download the source code from our Subversion server and run it. For users that require a more friendly installation package, we make nightly automated "builds" of all branches for a number of operating systems. For most platforms this is basically an installer to go with the source. For Windows, we also put the Slimserver source in a "wrapper" so it can be run as a windows executable on systems without perl installed.

At the moment, there are two branches in our revision control system. The trunk is being called 7.0, and there is a branch called 6.5.2. 6.5.2 is very close to release, so we are trying to minimize the changes that go into it so that we don't introduce new bugs. Nightly builds are available. The builds from the trunk (7.0) are not yet available on the nightly web page because there are still many known problems. It would not be constructive to have many end users filing duplicate bugs at this point. After it becomes more mature, we'll link to the nightly builds.

If you file a bug you are having with 6.5.1 (or an earlier version), one of the first things we'll do is ask you to try it on 6.5.2, which at the moment is quite stable. If the bug also exists in 6.5.2, we'll try to assess the severity and risks associated with fixing the bug in the 6.5.2 branch. If there's a risk of creating or exposing new bugs, we'll probably decide to mark the bug to be fixed in 7.0. This is not an absolute decision; if there's a lot of interest in having it fixed for 6.5.2 by all means let us know and we'll do what we can.

There have been some recent posts about how there's not much development going on with Slimserver at the moment. The main thing that's really happening is in the "trunk" of our source tree, there is under way a complete rebuild of error and status reporting between different portions of Slimserver that should allow much better error messages and error logging.

There have also been some questions regarding bugs filed. I believe you if you say that you saw a bug in 6.5.1! However 6.5.1 is done. That branch is now called 6.5.2! So if your bug is fixed in 6.5.2, then that's where it's fixed. We're not going to make any more 6.5.1 builds. Similarly, if a bug has been fixed in 7.0 but still exists in 6.5.2, it has probably been fixed by the major changes that have been done on 7.0 and it may not be possible to fix it in 6.5.2. Ever! If we were to make those same major changes to 6.5.2, then it would be 7.0, and you'd have to wait for it, too. :-)

If you file an enhancement request, what's going to happen is it will get marked to fix for a "Future" release. We already know what our goals are for 6.5.2 and 7.0. If your request is extremely important, or happens to correspond with work we're already doing for those branches, it might make it in. Otherwise we'll review it in a meeting later to determine what version we want to put your request in to.

Hopefully, this post about the background of software engineering going on with Slimserver will explain some of the strange behavior you see from developers and QA people! But we haven't all been replaced with policies and automation. If there's something you have a strong opinion about, always feel free to make a comment or send some email!

MrSinatra
2007-02-23, 14:14
that was an excellent breakdown, thx.

personally, i think each bug that is filed should garner a response from a human being at SD. i don't like it when i file a bug, and no one from SD responds, even to say u are now aware of it, let alone ask questions about it to flesh out what i didn't include (b/c i didn't think of it most likely).

i realize u all probably get an email from each bug filed, but the human touch goes a long way.

it would also be nice if after a few days of filing a bug, a response from SD would come about whether or not they were able to reproduce it.

just my opinion. let my crucifixtion begin...

haunyack
2007-02-23, 14:17
that was an excellent breakdown, thx.

personally, i think each bug that is filed should garner a response from a human being at SD. i don't like it when i file a bug, and no one from SD responds, even to say u are now aware of it, let alone ask questions about it to flesh out what i didn't include (b/c i didn't think of it most likely).

i realize u all probably get an email from each bug filed, but the human touch goes a long way.

it would also be nice if after a few days of filing a bug, a response from SD would come about whether or not they were able to reproduce it.

just my opinion. let my crucifixtion begin...

Thought police have you in their sights now! ;-)

slimkid
2007-02-23, 15:21
Hi Chris,

I'm asking this in a good faith, so that people don't jump over me just for being a party pooper.

So, the question is:

What is exactly your (I mean slimdevices' )definition of 'bug'?


K

ChrisOwens
2007-02-23, 16:31
It is our goal to have every bug report get a human response, but we have gotten a bit backlogged. We've hired some people (with Logitech's money :), and we're working our way up to the present. Please bear with us. However, as I describe below, if your goal is getting prompt and effective support, the bug database is not where you want to be.

As for the definition of a bug, I think it's like the Supreme Court's defition of obscenity. I know it when I see it.

But seriously, a bug can be nearly any problem with our software or hardware.

One persistent problem for QA, though, is that many users report bugs that turn out to be really support issues. We'll try and help, but especially if you want a prompt answer, you're almost always better off contacting support@slimdevices.com. Their job is helping you fix *your* problems and they're really pretty good at it. QA's job is finding problems, making sure they're correctly documented with step-by-step instructions for how to reprodce, and assigned an initial priority.

If a user files a bug that says 'I just downloaded 6.5.1 and it doesn't play any kind of music', you're probably going to get a response that says 'It works for me, can you give me more information?' whereas if you contact support, they will walk you through troubleshooting the problem on your system with the goal of making it work.

It definitely happens that after contacting support, a problem will get filed as a bug. There's no problem with that.

Fairly often, a bug turns out to be different than what the user reported. For example, a bug might be reported initially that some tracks aren't getting scanned but the 'real' bug might be that the installer didn't give some installed files the correct read/write/execute attributes. Sometimes we call the underlying bug the 'root cause'. Another thing that often happens is a bug will get marked as a 'Duplicate' of a bug that seems completely different. Usually this is because we happen to know that the bug in the underlying structure of Slimserver is causing both symptoms, and we're pretty confident that the 'fix' for the original bug will also fix yours.

A lot of times our initial replies to bugs seem to be interpreted that we don't believe the bug report. I believe you! But it's a more effective bug fixing process if we are be able to reproduce it. The developer can watch it happen in detail, and formulate the exact best fix and QA can evaluate the fix to make sure it worked.

I hope this answers some of your questions about bugs!

erland
2007-02-23, 23:57
Christopher, some thoughts:

1. Some ideas to make the long bug list today easier to use
- Maybe the "Target milestone" field could be part of the list result, today you can see STATUS in the list result but that doesn't tell me which release a bug/enhancement is scheduled to be corrected in.
- Maybe it would be good to have a easy way to search for "just open bugs" or "just open enhancements", today you have to use the advanced search mechanism to do this.

2. When we are talking about feedback on bugs I have often felt that reporting a bug is like putting the problem in a box which won't be opened for a long long time. One thing that would probably be a lot better even though its probably isn't so much work would be if the process was:

Step 1: A bug is registered by the user: STATE=NEW
Step 2: SlimDevices/Logitech reads the bug checks that the information provided seems to be enough, adds any additional information and changes the state or some other field to indicate the the bug has been recieved. Maybe also give some prelininary indication if the bug will be corrected in the maintenance release (6.5.2), the current development release (7.0) or in some future release.
Step 3: A notification is sent from the bug tracking system to the user that registered the bug.
Step 4: Some time passes by
Step 5: Some developer starts to work on the bug, reproduces the problem and solves the bug.

I think the critical feedback is at step 3, my experience so far is that that indication can take a long time. The reason is probably because you are trying to do too much investigations before giving feedback. As a bug reporter I would be happy with just some sort of indication that someone has seen my problem, this would also increase my interest to report more bugs in the future.

3. Do you want regular users to report bugs, or is it better that they contact regular support with their problems ?

4. As you probably already is aware of people are sometimes using the discussion forums for support issues when they ought to contact the support organisation instead. Maybe it would be a good idea to have a sticky thread or something in the discussion forums that describes when you should contact the support organization instead of the discussion forum ?
I guess this is basically one thing you are trying to do with this thread.

5. What about correcting bugs ?
If you have reported a bug and it is planned to be corrected in a future release. If the bug reporter is a developer, would a patch for the bug attached to the bug report result in that the bug can be included in an earlier release ?
If a patch is provided some time after the bug was reported will this be automatically be picked up by someone at SlimDevices/Logitech or is some other type of notification also required for the patch to be noticed ?

Skunk
2007-02-24, 12:21
I hope this answers some of your questions about bugs!

I'm becoming arachnaphobic, but have one possibly absurd idea.

You mentioned false/misused report, and one phenomenon I've noticed around here is that users are quick to point out mis representations of buggy behavior in the forums, but (for me at least, and from the web) bugzilla is more clunky than one of the forums.

Not sure if requiring a separate account sign up is a control feature, notification aid, or feature developers prefer, but a moderated forum seems more friendly to browse, from an outside perspective.

JJZolx
2007-02-24, 13:44
- Maybe the "Target milestone" field could be part of the list result, today you can see STATUS in the list result but that doesn't tell me which release a bug/enhancement is scheduled to be corrected in.

After you run a search, use 'Change Columns' to customize the data shown. This customization is saved to a cookie and will be used for all of your future searches.


- Maybe it would be good to have a easy way to search for "just open bugs" or "just open enhancements", today you have to use the advanced search mechanism to do this.

If you're logged in, then after using the Advanced Search to run a custom query, you can easily save it using 'Remember Search as...'.


2. When we are talking about feedback on bugs I have often felt that reporting a bug is like putting the problem in a box which won't be opened for a long long time.

Same here. Or a box that's never opened. It's commonplace to see bugs linked to in the forums when somebody mentions a problem, then find that the bug has never received a comment or followup of any kind up by anyone at SD or any of developers.

MrSinatra
2007-02-25, 11:29
another appreciated response, however at this point, i feel it necessary to give you some feedback you may not like, but hopefully is considered constructive:


It is our goal to have every bug report get a human response, but we have gotten a bit backlogged. We've hired some people (with Logitech's money :), and we're working our way up to the present. Please bear with us. However, as I describe below, if your goal is getting prompt and effective support, the bug database is not where you want to be.

but what if it is a bug, not a support issue?

the thrust of this post seems to be 'don't files bugs, call support.'

well, i have filed bugs and they were bugs, not support issues.


One persistent problem for QA, though, is that many users report bugs that turn out to be really support issues. We'll try and help, but especially if you want a prompt answer, you're almost always better off contacting support@slimdevices.com. Their job is helping you fix *your* problems and they're really pretty good at it. QA's job is finding problems, making sure they're correctly documented with step-by-step instructions for how to reprodce, and assigned an initial priority.

i absolutely can see where it could become overwhelming for the bug system when users take their genuine support issues there, which are not bugs. i definitely see that.

but this just gets back to the human element... if when a user files a bug, a human were to read it at SD and respond, i'd bet most of the mischaracterized bug reports could be filtered out right then, and referred to support.

i know in my case, the 3 bugs i have filed are indeed bugs, and that doesn't include a 4th bug that i did try to take thru support, with extremely disappointing results. luckily, after about 6 months, i was able to make headway with andyg here on the board, but it is merely a workaround, the bug still exists. (more on that in a bit)


A lot of times our initial replies to bugs seem to be interpreted that we don't believe the bug report. I believe you! But it's a more effective bug fixing process if we are be able to reproduce it. The developer can watch it happen in detail, and formulate the exact best fix and QA can evaluate the fix to make sure it worked.

sure...

but what about when you can't duplicate it thru no fault of the user? yet it is a SD bug?

my jpg bug was just like that.

but what really disappointed me was the SD reaction to the net radio streaming bug.

when 6.5 (or whatever it was) introduced non-proxied direct streaming of net radio directly to the SB, it made my SB useless to me as i use it to listen to one stream 90% of the time i use it at all.

unless i'm mistaken, i was the first person to bring this issue to SD's attention, which i did first here on the boards.

as time passed, i got nowhere, and the constant focus on me, and my setup, my microwave, and everything else got ridiculous. i realize those items must be eliminated first, but when i had done so and it was time to move the focus from me to SD, i found it was IMPOSSIBLE to do so.

when i did eventually try support directly, the response was horrible. there is no other way to describe it. i can forward you the emails if you like, but it was no help at all. and i'd be shocked if they filed a bug based on my situation.

the situation, which i eventually gave up on as it was like banging my head against the wall, only got resolved when other board posters started posting the same issues i was having. at that point i jumped back in.

thats when andyg, (about whom i want to be clear, i have no personal complaints), enacted a workaround so streams could be forced to proxy. but that took at least a minimum of 6 months, and other peoples complaints, to enact.

in that time, i barely used my SB at all.

so going thru support, was a dismal deadend. i think you should be aware that bad experiences like that make people not want to even try support.

and it is also worth noting, that the bug still exists! but is there a bug filed for it??? i don't know...

the point is, i must now have a computer on to listen to my stream. this is not the way it should be. i should be able to listen to my stream without a computer on. but is there a bug filed or being worked on to fix this? i have no idea, and support did nothing to help.

so chris, i'm sorry if i sound like a crank, and again i want to be clear that i think andyg does the best job he can do given the circumstances, but having said that, SD does need to pick it up.

in closing i just want to reiterate:

1. a human response in 24 hours of filing a bug is really needed.

that would be a great time to refer bugs to support that SD likely considers support issues, not bugs.

2. after that, within lets say 5 business days or so, a response from SD should come saying they were, or were not, able to reproduce the bug, or that upon further examination, it is thought to be a support issue (should that turn out to be the case), or that more time is needed, (5 days at a time).

i mean all the above constructively, no personal insult intended.

thx, -mdw

ps. let my crucifixtion belatedly begin...

ChrisOwens
2007-02-26, 15:26
You're going to have to get a lot meaner for me to take any of this personally. :)

If what you have is a bug, then file it as a bug. There is no problem with that.

Since we have recently been growing the QA team from 1 (me) to slightly larger (me +3), what's now happening is after a bug comes in, it will be assigned to one of the QA staff and the user will be notified when that happens. If it's useful, we can also post a message to the bug at that time that says 'I'm now reviewing this bug.'

It's my goal to have every bug get a response (at least a notification that it's been assigned to a QA person) within a few of business days. It would be a major change for us to promise a response within 24 hours.

Bugzilla has controls for what types of users are allowed to change what fields. If a user can edit a field, I have no problem with them correcting a bug. If they can't edit a field, they should feel free to post saying 'This happens on all platforms, not just solaris' or whatever correction needs to be made.

With regards to our bug process, basically our existing process is exactly as erland described with small differences. The primary difference is I haven't had enough resources to actually implement it. Now we do, and we're trying to make sure those steps are actually followed.

To summarize:

1) Bugs come in
2) They get assigned to a QA person (notification is sent)
3) The QA person makes corrections, makes sure fields are filled out correctly, searches for duplicates or related bugs, and checks that there's enough information to try to reproduce.
4) The QA person tries to reproduce
5) The bug gets assigned to a developer or back to me if there's some kind of problem.

erland
2007-02-26, 18:16
Since we have recently been growing the QA team from 1 (me) to slightly larger (me +3), what's now happening is after a bug comes in, it will be assigned to one of the QA staff and the user will be notified when that happens. If it's useful, we can also post a message to the bug at that time that says 'I'm now reviewing this bug.'

....


To summarize:

1) Bugs come in
2) They get assigned to a QA person (notification is sent)
3) The QA person makes corrections, makes sure fields are filled out correctly, searches for duplicates or related bugs, and checks that there's enough information to try to reproduce.
4) The QA person tries to reproduce
5) The bug gets assigned to a developer or back to me if there's some kind of problem.
My feeling is that it is important that the notification is received within about 1-14 days after the bug has been reported. The ideal would really be if that were after step 4 but I feel that it might be a bit to optimistic to think that all bugs will be reproduced within 14 days. It maybe can be realistic that step 3 could be perfomed within 14 days and it would then be a good idea if the notification could be after step 3. A notification at step 2 doesn't really say me that someone has looked at the bug so I feel that it would be a good idea if the notification could be after step 3 instead, at least if you think it will be performed within 14 days after the bug was reported.

I also have a feeling that there might people who feels that 14 days is really long time, so the notification limit maybe shall be 7 days not 14.

I think it would help if the notification could be as a posted message in the bug report which everyone understands even though they done understand the whole bug handling process itself, for example "Reviewed by QA responsible" or something. In my opinion the notification should indicate that something has been done not that you are starting to do something, this is also a reason why a notification after step 3 feels better than a notification after step 2.