Hi All,
baloo is currently annihilating my disk to the point where I'm getting frequent UI freezes of a couple of seconds long every minute or so.
I'm running Kubuntu 14.04, up to date as of ~ 1hr ago
The disk activity led on my computer has been hard lit for the entire hour, and baloo_file_extractor is making consistent appearances in the process list

Google didn't turn up a bug on the topic, and other issues seem to relate to high CPU usage (which I'm not seeing, only disk usage).

I followed the migration path from nepomuk laid out in the sticky thread.
I don't want to disable search indexing because I find it handy -

Any recommendations, or should I head to the bugtracker?

PLease check with "iotop" or similar tools what is the cause of I/O. If it is "baloo_file_extractor" please note a few of the numbers next to it, and try issuing

Code: Select all

balooshow <number>

Where <number> is one of the numbers you noticed earlier: see if they're particular file types (big, etc.).

einar wrote:PLease check with "iotop" or similar tools what is the cause of I/O. If it is "baloo_file_extractor" please note a few of the numbers next to it, and try issuing

Code: Select all

balooshow <number>

Where <number> is one of the numbers you noticed earlier: see if they're particular file types (big, etc.).

That's pretty handy

Ok, so the issue is, this is a work desktop. I'm connected to multiple network drives via the automatic mount.cifs/pam_mount.conf.xml config, and the mountpoints for these are located in $HOME.

For the moment I have removed them from the search list via the system settings -> desktop search dialog, and that seems to have helped a fair bit (the file cleaner is topping the iotop list now). I'll give it an hour or two to calm down while I'm at lunch.

Is it worth me reporting a bug on the overly high IO for networked filesystems? The impact on performance it was causing me is not what I'd consider 'normal'.

The other interesting thing is that there were two automatically added entries in the desktop search exclusion list, which were other local filesystems (an SSD, and an old ubuntu partition). Perhaps *all* external filesystems should be automatically added?

baloo_file_cleaner still going ...

Hi.

Yes, perhaps we should be disabling all external media by default, no matter where they are mounted. Would you mind filling a bug for that? I can possibly even get it into the 4.13 release.

How bad was the IO usage? Cause I've spent quite some time trying to optimize it, but it's clearly not enough. I'll look more into it.

Regarding the file cleaner - Yeah, that hasn't been tested much. But it should not be going on for 3 hours! Is it consuming tons of cpu / io?

Personally I've seen it eat a lot of RAM, more than I/O, but I admit I had removed directories with really huge files from the folders to be indexed.

vHanda wrote:Hi.

Yes, perhaps we should be disabling all external media by default, no matter where they are mounted. Would you mind filling a bug for that? I can possibly even get it into the 4.13 release.

How bad was the IO usage? Cause I've spent quite some time trying to optimize it, but it's clearly not enough. I'll look more into it.

Regarding the file cleaner - Yeah, that hasn't been tested much. But it should not be going on for 3 hours! Is it consuming tons of cpu / io?

Bug submitted: 333433 (gutted I missed 333333 )

"How bad?" is difficult to quantify, in terms of practical usage performance hits it was significant. Switching tabs in chromium would take seconds at a time, apt-get operations that required any dpkg database access took magnitudes longer (eg I'd be waiting 1 or more minutes for an apt-get update just to read the package lists which is normally a 3s affair), typing emails in a web interface would frequently pause for 1-3s, showing/hiding my yakuake console, and switching terminals would often take seconds.

Re file cleaner cpu usage, I didn't look at the cpu usage explicity. IO usage was registering on iotop but not causing a performance hit, I'd guess the same went for cpu usage (ie it would've been registering, but I wasn't being adversely impacted).
I can run it all again when I have some downtime, I had to take yesterday off work and am catching up a bit

I'm having the same problem. Disabling baloo makes everything normal. With it running things are slow and get slower to the point that the system is no longer usable (waited 10 minutes to log into a console yesterday before I forced a crash).

I did the iotop and numbers thing. It was working on files that were not so large, but in directories that have many many directories. It is ebooks so something like:

/home/alw/Calibre Libraries/Lib1/AlW/cover.jpg

For example, Lib1 has almost 18,000 subdirectories in it.

File system is ext4. It is an LVM2 volume on md RAID. EnhanceIO is running but turning that off does not help. BFQ scheduler (although problem observed with the default scheduler as well).

I think it is the giant number of subdirs but that's just a hunch.

Edit. Oh yeah, file system options:

Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: journal_data_writeback

Note dir_index is on and the journal is in writeback.

The large number of directories is quite likely the cause. If you exclude this eBook file structure, does the performance improve?

It certainly helps, but not completely. The system is visibly sluggish. For example, chrome will show "waiting for cache" significantly when loading a page. One difference is that before the culprit was the extractor. Now only baloo_file and baloo_file_cleaner seem to be running but they are consistently at the top of iotop and noticeably starving the system.

The file cleaner is likely responsible for this - see https://bugs.kde.org/show_bug.cgi?id=333807
If you don't mind the other directories being reindexed freshly, it may be more optimal to drop the current set of indexes (outside KDE).

That's a pretty amusing bug... However, even if I drop the indicies and go again, it assumes that I've excluded everything and have <1000 files, right? Even with the exclusions I asm pretty sure I have more than that. Does the cleaner only fire if the files are deleted? If so, how would that help as even with the big directories I would probably not be deleting thousands of files.

In 4.13.1, the cleaner will simply drop the database and reindex according to the preferences the user made, and thus should sidestep the bug.

In a fit of boredom, I wrote this quick hack: https://github.com/wd5gnr/balooctl

Yes, it should use QtDbus. No, it doesn't (yet).

And the developer just added a "balooctl" command. It should be available in 4.13.1.