PDA

View Full Version : Virtual tracks to remove duplicates.



oreillymj
2005-11-08, 07:40
Hi,

I'm not sure if this has been asked before, but I'd like to remove a lot of the duplicate tracks in my collection and only have 1 copy of each file. But I'd also like to have a virtual copy of certain tracks in my collection.
I know I can achieve kind of what I want with playlists, but not exactly.

e.g
I have 4 U2 albums with all tracks ripped to my library.
I also have U2 greatest hit which replicate most of what's on the 4 albums.

I'd like to only rip the new tracks off the greatest hits, but have pointers to all the other tracks using the existing tracks off the 4 albums. So when I view the U2 greatest hits in the UI, I'd like to see all the songs that are on the CD case whether they exist as real files or not, becuase those songs exist somewhere in my collection.

I tried using a Windows shortcut which didn't work.
Then I tried using 1 track playlists, but they only show up in Browse Playlists" and do not alter the list of tracks displayed in the UI for an album.

Anyone else trying to do this or sucessfully done it.

Steve Bernard, Jr
2005-11-08, 09:41
On 11/8/05, oreillymj <oreillymj.1y6ncz (AT) no-mx (DOT) forums.slimdevices.com> wrote:

> I'm not sure if this has been asked before, but I'd like to remove a
> lot of the duplicate tracks in my collection and only have 1 copy of
> each file. But I'd also like to have a virtual copy of certain tracks
> in my collection.
> I know I can achieve kind of what I want with playlists, but not
> exactly.
>
> e.g
> I have 4 U2 albums with all tracks ripped to my library.
> I also have U2 greatest hit which replicate most of what's on the 4
> albums.
>
> I'd like to only rip the new tracks off the greatest hits, but have
> pointers to all the other tracks using the existing tracks off the 4
> albums. So when I view the U2 greatest hits in the UI, I'd like to see
> all the songs that are on the CD case whether they exist as real files
> or not, becuase those songs exist somewhere in my collection.

I thought about doing this in Linux with symlinks, but then I realized
the file tags would be all wrong. Like, the album name would be
tagged incorrectly, and most likely so would the track numbers of the
individual songs. I suppose you could use file and folder naming and
have it guess that info, but even then you'd be out of luck with more
sophisticated tags.

However, if that's good enough for you to have Slimserver guess based
on directory placement, you might want to check the XP Resource Kit
for a tool called fsutil (assuming you have Windows XP and the hard
drive is formatted with NTFS) and use that to make hardlinks of those
files. The syntax would be like:

fsutil hardlink create "c:\artistname\albumname\04-Song Name.mp3"
"c:\artist\Greatest Hits\01-Song Name.mp3"

I'm not sure what Slimserver does with hardlinks in Windows (I don't
run Slimsever on my Windows machine anymore), but it might work. In
my experience, apps have no idea that hardlinks aren't completely
separate files. It could be worth a try.

If anyone has any suggestion about file tagging using linked files
(maybe cue sheets would work a bit better), I'd love to hear it, but
I'm not holding out much hope for that.

-Steve

Ben Sandee
2005-11-08, 09:57
On 11/8/05, oreillymj <oreillymj.1y6ncz (AT) no-mx (DOT) forums.slimdevices.com> wrote:
>
>
> I'm not sure if this has been asked before, but I'd like to remove a
> lot of the duplicate tracks in my collection and only have 1 copy of
> each file. But I'd also like to have a virtual copy of certain tracks
> in my collection.
> I know I can achieve kind of what I want with playlists, but not
> exactly.


This has been discussed previously -- it pretty much devolved into a two
bitterly divided camps. Check the archives and you can relive the fun.

There are technical problems beyond the tagging issues -- for example often
the tracks themselves have been remastered on these types of collections.
Normally the tracks are *at least* volume normalized so while you may think
the tracks are identical, they in fact are not. Disk space is cheap...
really cheap! The effort it would take to do this properly just makes it not
worth the effort for most people. Certainly I would HOPE that SlimDevices
has better things to spend their time on.

Ben

Bart
2005-11-08, 09:58
you might want to check the XP Resource Kit for a tool called fsutil (assuming you have Windows XP and the hard drive is formatted with NTFS)
I have done this and it works seamlessly (except for the track number problem)
It is also good for tracks featuring two artists, you can make the track appear under both artist's names.
However I subsequently put a bigger disk into the system and copied the files across whereupon NTFS converted the links into files (because the links only work on one physical volume) Now I have some unknown quantity of duplicates in my system and have hesitated to create more links because I am bound to upgrade the disk sooner rather than later.
Anyone got a useful duplicate finder?

Ben Sandee
2005-11-08, 10:04
On 11/8/05, Bart <Bart.1y6tlz (AT) no-mx (DOT) forums.slimdevices.com> wrote:
>
>
> Steve Bernard, Jr Wrote:
> > you might want to check the XP Resource Kit for a tool called fsutil
> > (assuming you have Windows XP and the hard drive is formatted with
> > NTFS)
> Anyone got a useful duplicate finder?
>

MusicMagic mixer has a duplicate finder capability (along with super-sweet
integration with SlimServer) -- it actually analyzes the music data to do it
so it seems to be fairly reliable -- a few false positives I've found. Note
that I don't do anything with the duplicates myself -- I'm not really
worried about the extra space myself.

http://www.predixis.biz/Predixis_Mixer.htm

Ben

oreillymj
2005-11-08, 10:06
iTunes has a good find duplicates function.

Steve Bernard, Jr
2005-11-08, 10:33
On 11/8/05, Bart <Bart.1y6tlz (AT) no-mx (DOT) forums.slimdevices.com> wrote:
>
> I have done this and it works seamlessly (except for the track number
> problem)
> It is also good for tracks featuring two artists, you can make the
> track appear under both artist's names.
> However I subsequently put a bigger disk into the system and copied the
> files across whereupon NTFS converted the links into files (because the
> links only work on one physical volume) Now I have some unknown
> quantity of duplicates in my system and have hesitated to create more
> links because I am bound to upgrade the disk sooner rather than later.
> Anyone got a useful duplicate finder?

Not this this would help your current duplicate situation, but the
Resource Kit also includes a program called hlscan.exe that goes
through and tells you all the hard links on the volume so you can
remove them and add them back later on the new volume. Of course that
doesn't sound very fun, either, though I suppose it could be automated
with a script.

-Steve

Bart
2005-11-08, 14:06
the Resource Kit also includes a program called hlscan.exe that goes through and tells you all the hard links on the volume so you can remove them and add them back later on the new volume
Thanks - that is exactly what I need. And I use Musicmagic too (and it is fantastic), so I shall investigate it's duplicate finder.

oreillymj
2005-11-09, 00:15
How about this as a suggestion.

With an option - "Enable Virtual Tracks" enabled.
Playlists in the playlist folder are treated as normal.

Playlists within the music folder (in an album dir) as treated as virtual tracks for that album.

Comments could be added into the m3u file to override the information inherited from the tags in the real file. Default behaviour is to inherit all info from the real file.

e.g. I have a Massive Attack album.
I also have several Chill Out compilations with Massive Attack tracks. I create a playlist in the ChillOut Album dir.

#EXTM3U
#<SD_VIR_TRACK>
#TRACKNO=1
#ALBUM=Chill Out Volume 1
#YEAR=INHERIT
#</SD_VIR_TRACK>
\Documents and Settings\Michael\My Documents\My Music\MP3\Massive Attack\Protection\04 Weather Storm.mp3
#<SD_VIR_TRACK>
#TRACKNO=6
#ALBUM=Chill Out Volume 1
#YEAR=INHERIT
#</SD_VIR_TRACK>
\Documents and Settings\Michael\My Documents\My Music\MP3\Massive Attack\Protection\05 Spying Glass.mp3

In this case the Album name and track position are being overridden. The year is being inherited from the year of the Massive Attack album, not the compilation, but this the defaul behaviour anyway.
The comment tag format would have to be documented. It could also be optimized to make it easier to parse.


Feedback?

Should I log an enhancement request?

Bruce Hartley
2005-11-09, 00:44
While we are heading off into the blue sky, what about this..............


In the "greatest hits" folder you have a very short MP3 file. (or flac or whatever).

It has no music in it.

It does have the tags that are relevant to that instance of the file, then a "new magic" tag, that points to the actual file to use for the audio portion.

Doable ?

oreillymj
2005-11-09, 01:51
"you have a very short MP3 file" ..suggests a binary file.

How would users create these binary files? How do they add the special tags to tell the Slimserver that they actually point to a file somewhere else? How do "native" players on the PC (like winamp) react when they hit these files?

My suggestion is really just an alternate special use case for the existing playlist functionality with some added comments in the file. Winamp etc. will completely ignore the comments and play the track(s) as if it were a normal playlist file.

Thinking futher about the data held in comment fields, it makes sense to use XML. There must already be a parser in place for iTunes.

Bruce Hartley
2005-11-09, 02:02
How would users create these binary files? How do they add the special tags to tell the Slimserver that they actually point to a file somewhere else? How do "native" players on the PC (like winamp) react when they hit these files?

I wasn't suggesting an implementation, simply an alternative blue sky suggestion.
I agree that your playlist suggestion is quite a clean one.

However at the end of the day, I have had thoughts of removing duplicates on greatest hits albums before and decided it would be way more trouble than it is worth.

oreillymj
2005-11-09, 03:38
Yes, I'm finding managing a collection of about 7000 tracks is becoming a chore. Duplicates, missing/incorrect tags etc....

But then, so is finding a particular CD among the 1000's I have spread around my house,car, wife's car.

Last time I fired up iTunes I had nearly 200 dupes. 200 * 6mg (avg) is 1.2Gb. So that's quite a long of wasted space, that I'd like to reclaim.

Bruce Hartley
2005-11-09, 03:46
I reckon your wasted 1.2Gb is less than 50 pence worth of disc space.
Is that really a big deal.

How much is your time worth, when your trying to save that 50 pence.

oreillymj
2005-11-09, 04:45
In terms of cost of disk space, it probably has low ROI.
But in terms of keeping my machine healthy and free of clutter, it probably is worth the effort.

Most of the code to do this probably already exists. Although there would probably need to be some additional binary field added to the Db schema to mark tracks as virtual so they are excluded from track counts etc.

radish
2005-11-09, 07:04
It's an interesting idea, but not one that I would use, as even though I have many instances of the same track they're slightly different in most cases. As someone who used to run a band website, keeping track of all the different versions of a given track, even with exactly the same name, was a pain.

And I'd love to know how removing a couple of gigs of data from a HDD keeps your computer "healthy" :)