After some years of slight disappointment and wondering if it has anything to do with my wifi or any other network issues, I finally figured out that the sync on the Controller itself just doesn't work that well. It works fine with any other combination of players (real SB, raspberry, SqueezeWRT), but still goes out of sync on the Controller.
Now I discovered that the wiki actually mentions this and there are some old threads (2008) on the forum discussing this when the 'Controller as a player beta' function was just released. Those post point to SqueezePlay being very new, but by now it has been around a while and so I am wondering if there has been any improvement since then?

Alternatively, it seems that the sync does work better with some music. Has anyone figured out if there is a relation to bitrate or codec? If so, would it help to force transcoding on the server?

Controllers come pretty cheap these days and they are fully contained systems with buttons and a screen. It's a bit awkward with the wire coming out on top if you want to attach speakers, but that's a matter of getting used to.