Rengels, my VA albums will all be found in ~/HOME//Music/Various Artists/Album/track

I always wondered why that was not a factor in the scanner deciding whether or not to sort into VA.

I do understand that tags are a better way, but in the early days I didn't know anything about tags. And my favorite tagger doesn't seem to reliably set an Album Artist tag, probably because I'm not skillfull enough in its use. Anyway, it seems that Various or Various Artists in the file organizations should be a fall-back, if nothing else.

the issue isn't the timing of "incremental scanning" if you're referring to the manual "update collection" but the time/cpu used accomplishing "watch folders for change" every 30 second or so - in the old scanner there was no notice of the watch process, it just happened totally without any noticeable system hits. Why isn't the "watching" accomplished with inotify? Wouldn't it accomplish the same without using the resources?

"find . -type d >/dev/null" takes 22 second for me when run at the top of my music directory and I have a much simpler directory structure:
~/MyMusic/source/<artist-album>/<artist-album-track-title> note: there are 5 source dirs

I suggest the "various artist" discussion be split into a separate thread from the scanner cpu/timing discussion

Exactly like google01103 writes, the issue isn't the timing of "incremental scanning" but the time/cpu used accomplishing "watch folders for change" every one minute for me. I can confirm in the old scanner there was no notice of the watch process. My 2 cent are: and if you delay the scan process? the CPU load can decrease? I think I don't care if the process takes seven or thirty seconds...

Regarding find command:
/media/xhdd/Music$ time find . -type d >/dev/null

real 0m10.485s
user 0m0.060s
sys 0m0.640s

P.S. in my case compilation albums are in /media/xhdd/Music/V/Various Artists/<compilation name>/<track no.> - <trackname> . <suffix>

Open Amarok > Settings > Configure Amarok > Collection. Untick "Write statistics to file" if ticked. This should solve the 30 second wake up issue.

Inodes or the kde file change watch or the mechanism that Qt is using for the file dialog.
All of those would be a better solution.

However my main target was to clean up the collection scanner and not how it was started. It took enought time to get even that in a sensible state.

There are some things the old scanner did different but frankly. If it takes ten minutes to just iterate over your whole collection, how do you expect the incremental scanning to be much faster?
And if it takes ten minutes, why are you surprised that it seems to be constantly scanning?

I was thinking about a better algorithm. After a scann wait at least last-scan-time * 200
That would keep the total load below 0.5%.


Valoriez: Directories are mostly ignored. I have a "Soundtracks" folder. Is this now the artist or the album title? The scanner can't know.
Just use Amarok to set the album titles. You can even use the "guess from file-name" feature. They will be stored in the tags and you should be set.

rengels -

the "guess from file-name" feature seems to have been neutered as it's only available for single tracks (and on my system it is greyed out even then), if multiple tracks are selected for editing only get tags from musicbrainz is available

"And if it takes ten minutes, why are you surprised that it seems to be constantly scanning?" - because this is a regression

Rengels, point taken about directories.

About the Album Artist tags -- I did set them in Amarok, and my VA are all being sorted correctly. It's the albums which are NOT VA, but do have various artist *tracks* -- classical works -- which are being set incorrectly now. I've got the Album Artist as Yo-Yo Ma, for instance, and the tracks are correctly attributed to their composers. The MusicBrainz standard for classical music is that the Artist is the composer, not the person playing the main instrument, the chorus singing the works, etc. However, the AA tag is ignored, and the tracks are sorted by composer who is tagged as Artist.

So our present method is working well for popular music, but not classical. Album Artist isn't being respected in the non-VA part of the collection, and I feel that it should be, and would solve this particular problem. I don't see why anyone would want the tracks sorted by composer? One can easily do a search for that.

@bcooksley:

Open Amarok > Settings > Configure Amarok > Collection. Untick "Write statistics to file" if ticked. This should solve the 30 second wake up issue.

????Great idea, I can also close amarok and use mpg123. I won't have collection issues anymore. Apart from that.. it is not set.

1. Structure was defined by myself ages ago, long before amarok came to life. Can't see, why this should be a problem. The scanner doesn't do any searches.
2. And what the hack are you talking about, regarding the compilations. I'm not talking about compilations. I'm talking about one album with all tracks in one directory with the same album tag and artist tag. Can't be so hard to detect.
3. Most compilations are under v/Various/<albumtitle>
4. As already stated above album artist is not set on single artist albums. Don't know whether it is set on all or some VA albums.

An album should only be a compilation if the album artist is empty or "various artists" or something similar.

. Incorrect. An empty album artist doesn't imply a compilation.
5. Old scanner needed some seconds to recognize the changes and some seconds to update the db.

Summary:
The old algorithms worked, new ones don't.
The new scanner runs without cause most time, the old one only when it makes sense.
The new one consumes more resources than the old one.

FAIL!

m0nk

valoriez. Can you send me a collection scan of such an album. I've seen problems before but I was always able to explain it.
Just send it to my private e-mail ralf-engels@gmx.de

Same for Dieter. Just send me the scan.
I've seen scans before where the tracks did have the same artist tags but some had an album artist set.

to point 3. No. they are not. They are under cds/Bad (with audio comments from Quincy Jones) or under new/Transformers.
You really can not go by the directory layout.

to point 4. Yes it does. If an album artist is set to an empty value that is the same as one set to "various artists". Both mean "I don't have an album artist". Only if there is no album artist tag, then we have to go by artist alone.

Please get your facts right.
- The old algorithm completely ignored album artists and the compilation tag.
- The old scanner did run as often as the new one does, every 30 seconds
- I don't know what the old scanner did but if it runs much faster than a simple find does, then it clearly didn't check all the directories.

Before writing FAIL, start up your editor and implement a nice inode watcher for Amarok. Amarok never had this but it's clearly needed.

sorry if I repeat myself, the real problem that I felt is that every minute for 4 or 6 or 8 seconds my cpu is close to 100% with significant impact on system performance during those spikes.

Okay, have filled in the album artist in the Ufomammut/Alan Parsons Project mix up for Ufomammut and this Eve album is now filled under Ufomammut (without cover). So the problem is the album artist tag, which has a higher priority as artist, although album artist is less important.
Some stats for the cpu issues:

ds@morpheus:/media/Mucke> time find -type d>/dev/null

real 0m1.867s
user 0m0.064s
sys 0m1.795s



m0nk

joethefox, you probably mix processor load with IO load. I also did this not too long ago.
There is no way around checking the modification date of every single directory. You even have to be carefull with doing shortcuts because you could miss symbolic links.

That reminds me. Do you have symbolic links in your collection? I've made some changes there.

Dieter, If someone set's the album artist then this is the one we need to use. So it is more important when it comes to putting tracks into albums.
There can only be one album artist per album.


Here are my times and I don't see a spike in my load but I still like to switch the watch folders off as it doesn't make sense to constantly watch for changes when the files only change once in a week.

real 0m0.034s
user 0m0.010s
sys 0m0.020s
(that was the second run when the directories are already cached)

@rengels
but looking at the System Load Viewer colors, the bar is 3/4 blu and 1/4 red, that should mean 3/4 user and 1/4 system according to standard Viewer's color (cpu usage growing similar to 30% > 50% > 90% > 0% in about seven seconds) , no IOWait (green bar). If you consider useful I'll try to record a cpu usage stats file.

@rengels,
I may have understood what my problem! I'm sorry if I've involved you. Probably indirectly the new amarok collection scanner showed a known problem: the filesystem where files are located is NTFS ( ) and mounted read / write (fuseblk) and almost 100% of my problem is related to ntfs-3g package and its way of managing disk access in special cases like maybe this one.