Like I said, I decided to enable Baloo a few days ago so I could use its file indexing abilities. It does eat CPU from time to time, but I guess I'm okay with that. There is however one bizzare problem which is a little worse:

Baloo seems to cause memory leaks, and permanently increase system RAM over time. The weirdest part is that this extra memory isn't attributed to any process I can kill. If I check the process list in KSysGuard, no extra process(es) eat(s) any extra amount of RAM. However, the hardware info charts show I have up to 1 - 2 GB more full memory than I should have.

This is particularly creepy since as far as I know, Linux (and other OS'es) cannot fill memory that's not attributed to any process. Memory leaks happen all the time of course, but they're always part of the process in cause. If memory leaked outside of a process and into the nothingness of RAM, I imagine that could even cause a Kernel panic or something bad. So what's happening, and does anyone else get this and knows how to fix it?

Filesystem cache?
File indexers should be prone to that (though i wonder how it could increase "over time")

The relevant information can be found in /proc/meminfo and - to a certain degree - the output of "free". NOT a process list - neither ksysguard nor top do a usable job on this.

Please also notice that virtual memory management is VERY complex and it's is probably impossible to inform users in some fancy numbers, since it largely depends on your interest in memory usage.

Iff the kernel would actually loose memory, that would be a severe kernel bug, but the chances that when a user thinks there's sth. wrong about memory management in fact the user simply doesn't understand it, are like 10.000:1.
(No offense, I trust the kernel more than myself as well and when I think there's sth. odd, I ask google and google tells me that there's still much to learn for me

This drops all file cache memory (run as root)

Code: Select all

echo 3 > /proc/sys/vm/drop_caches


The kernel will claim FS cache memory whenrequired by applications (depending on available swap partitions and some other settings in /proc/sys/vm/, mostly vfs_cache_pressure and swappiness)

Aha! Echoing 3 in drop_caches instantly clears up the excess memory... so at least I have a workaround. I hope this doesn't mean something bad like a Kernel fault.

I assume posting the content of /proc/meminfo is pointless now that the bad memory is gone, so maybe I'll do so when that happens again.

Either way, the problem only appeared after I enabled Baloo... so clearly it's relevant in some way. Maybe it's all the disk writing it's doing when it's saving the caches?

there's no problem, everything is fine and you should not be in need to drop the FS cache - the kernel will claim this memory whenever it needs it.
you can tweak "when it needs it", but desktop users should not have to.

if your only "problem" is that you do not like a number in a resource plasmoid/conky, you want that thing to only show "active" process memory (since that is what is really inavailable, used by all processes) - dropping the fs cache for no reason WILL SLOW DOWN THINGS. everything. (since the fs has to be re-read from disk on need and your hdd or even ssd is slower than the ram by the order of some magnitudes)

Yes. Baloo is probably the cause of this.

In order to track new files, deleted files and moved files we use a certain kernel system called inotify. Inotify is awesome, except that it really is not. It requires us to recursively stat every single directory which should be indexed. So on startup, your file system cache will get filled to a considerable extent.

Please note, that this isn't a problem specific to Baloo. It affects all file indexers in Linux.

I understand. So there's no way around the problem currently, but it also shouldn't be a practical issue and only seeing high RAM usage.

I noticed that for some reason, the more memory I have full the slower my system gets, even if it's far from the maximum amount of memory. So having 25% of my RAM full makes a small difference from having 50% filled, even if nothing uses the remaining 50%. Also, where does the indexer RAM usage actually stop? Could it go as far as leaving my computer without free memory for other applications and crashing it?

Either way, this sounds like a problem that needs to be fixed. But not by Baloo necessarily... rather by the Kernel in this case.

I noticed that for some reason, the more memory I have full the slower my system gets, even if it's far from the maximum amount of memory.

How did you measure that?
If it's a gut feeling, your gut is wrong (unless 50% of your memory are a swapdisk...)


Also, the kernel *will* drop all FS caches before crashing any process OOM.
If you're really running OOM otherwise (active process memory), you're screwed. The kernel will try to kill the culprit, but might hit random innocents.
This problem affects all OS and is basically unfixable (unless *you* come up with a patch

("best" is to have a small swap partition - this will slow down things so much that you can spot the memsucker and kill it yourself)

luebking wrote:How did you measure that?
If it's a gut feeling, your gut is wrong (unless 50% of your memory are a swapdisk...)

It's an impression I got over the months / years. When I first start up my machine, and only a little bit of my RAM is used, everything works super fast. After a day of uptime or more however, processes begin filling the memory a little bit more (permanently), thought to a normal and sane extent. At that stage, all applications also take a bit longer to work or start up.

I think what I'm noticing is the effect of memory fragmentation. Big slowdowns usually happen when I open a process that eats almost all my available RAM. Even after I close it, and the used RAM goes back down to a small value, everything still works very slowly for several hours because of how the memory of applications was re-arranged and mixed in. I imagine that the Baloo problem is one thing that can contribute to this also.

you can back your impression with some hard data

See https://www.kernel.org/doc/Documentatio ... s/proc.txt, search "buddyinfo"

Mem fragmentation should have become less of a performance problem since late 2.6 kernels (and should be no longer in 3.x kernels), though.

BTW, I made a Kernel bug report too if anyone's interested: https://bugzilla.kernel.org/show_bug.cgi?id=77411

In regards to the delayed performance - do you have a swap partition enabled? If you are using applications which run your system close to the memory limits then it is possible that parts of system and KDE processes are being swapped out. The delay you see is because they need to be swapped back in to run. You could also be seeing the delay due to the content they need not being in the filesystem caches the kernel maintains, requiring a trip to the disk to retrieve it (which will always be slower than retrieving it from the caches).

I have a SWAP partition, although my RAM is rarely filled to the top so it's barely ever populated.

Okay, it is probably just the reinitialisation of the cache then with the content your particular workload requires that is causing the delay. There is nothing Baloo can do to avoid this - any program which uses disk will cause this sort of behaviour.

bcooksley wrote:Okay, it is probably just the reinitialisation of the cache then with the content your particular workload requires that is causing the delay. There is nothing Baloo can do to avoid this - any program which uses disk will cause this sort of behaviour.

That sounds about right. I haven't noticed this until now because nothing I ever used reads / writes so much on the disk as file indexers. Or actually, I might have noticed similar symptoms when copying huge files across partitions, though in a much less drastic form.

I could however imagine a few things Baloo can do to improve this. Such as an user setting to limit the maximum activity allowed. So if someone feels that the indexer is causing too much disk activity, it limits it to 10 files per hour for example. Yes, this will mean that it could even take Baloo months to index a whole drive... but if that offers more care for system resources it might be worth considering.

There is another even better feature I could suggest: Allow disabling the automatic file indexer, and have Baloo add files to the cache when KDE applications that communicate with it access them. For example, when Dolphin enters a folder for the first time, it also sends the full file list to Baloo. This should result in nearly no extra disk activity and maybe no more CPU usage entirely! Of course, only files you access in a KDE program are ever noticed in this case... but then again, the user might not care to have files he never accesses indexed, and this can be optional.

Mostly @Vishesh:
Don't baloo indexers run ionice?