Hello Again!

Well, I made some tests yesterday and some adjustments and the situtation is much better now.

At first I disabled all the Semantic Desktop altoghether. But then I lost possibility to search through my mails. This was not user friendly solution, so I started to make some tests.

First, I re-enabled the Semantic Destkop, but without file and message indexing, at the same time I raised the memory Virtuoso is allowed to use from 50Mb to 512Mb, since I have 8Gb of RAM and lots to spare. After this I noticed that the laptop is still running very smoothly and if virtuoso-t process shows up, it goes to 25% and back to 0% very rapidly. It does not even register on the graph. So, increasing the amount of memory allowed definitly speeds up the Nepomuk database operations and I suggest to those with enough RAM to raise the RAM to higher value.

Still this all was without ANY indexing going on. So, I made the test and re-enabled the file indexing, but left mail indexing off. And here I found the culprit that was eating my resources. Once I enabled the file indexing, the heavy load returned and with the System Settings window open I was observing how the Nepomuk was indexing my files. I left things for an hour in hope that they would settle down, but no dice. Load was heavy, CPU graph was indicating heavy load. So, instead of switching off the file indexing altogether, I suspended it in the system tray. This way, I have the file search function, but without constant re-indexing of files.

Things settled down again after suspending the file indexing. So, I decided to re-enable mail message indexing to have the search function in KMail. Lo and behold, the Nepomuk did not put any overload on the system. I was able to use KMail without any delays.

So, to sum it up, I raised the RAM for Nepomuk virtouso-t process to 512Mb and suspended file indexing (instead of disabling it). Now, the system is stable, the CPU graph is flat like a line and I can finally focus on what I want to do, not on the tool itself.

On the side, I found out that akonadi_mixedmaildir_resource process eats up ALL my RAM with time and it does not release it. When I start KMail it uses 4Mb of RAM. When I start going down my folder list, it starts to literally swollow RAM until it reaches 1.6Gb. It never releases this memory, even after KMail is switched off. The only way is to log out of KDE or kill the process. Then it restarts and uses 4Mb of RAM again. So, if I do not browse through too many mail folders, the process is not using up too much of RAM. But if I use the KMail the whole day, searching for files in many folders, I end up blocking 1.6Gb of RAM.

I do not know if it is a bug or the reason I have so many e-mails and files and the whole indexing is causing it, but I suspect it is akonadi related bug. It seems like akonadi takes the whole index of currently viewed folder into memory and keeps it there, instead of releasing it after viewing was finished.

One more thing to worry about...

@szob: Approximately how much mail did you have indexed?

bcooksley wrote:@szob: Approximately how much mail did you have indexed?

Currently, above 130,000, plus above 46,000 files indexed in my home directory. Size of the Virtuosobackend database is 1.2Gb.

Any other info?

Now, after the tweaks everything is working as it should. CPU vent barely works and battery is running long. KMail is also much more smooth, but I had to go to the top KMail folder and disable ANY synchronization, since each time I stood on the top folder it would run synchronization for akonadi and needless to say synchronizing 130,000 mails took 3-4 minutes. I only left synchronization of folders when I highlight them. So, for those folders I do not view, synchronization is off.

I upgraded to KDE 4.8 (from 4.7.x) yesterday and since that time I have virtuoso problem too. I had this problem after switching to Kmail 2.x, then after a lot of effort (basically starting everything from scratch going step-by-step with small amount of emails re-adding to Kmail) I was able to make it stable not eating all my CPU power. And it is back again. I have already set memory to 512MB and still virtuoso eats 100% of CPU. I am so frustrated by these problems in KDE

KejPi wrote:I upgraded to KDE 4.8 (from 4.7.x) yesterday and since that time I have virtuoso problem too. I had this problem after switching to Kmail 2.x, then after a lot of effort (basically starting everything from scratch going step-by-step with small amount of emails re-adding to Kmail) I was able to make it stable not eating all my CPU power. And it is back again. I have already set memory to 512MB and still virtuoso eats 100% of CPU. I am so frustrated by these problems in KDE

Well, you can always switch off the Desktop Search in the System Settings and the problem is solved, BUT you will not be able to search for mails or files.

Now, there could be a few reasons why you have the problem. Firstly, the Nepomuk may be re-indexing the content of your home directory and in my case it took 10 hrs. So, leave your PC for a night to work it out.

In my case, the Virtuosobackend got some database error and went into the loop thus using 100% of CPU. That's why I was forced to delete the entire index database which was 1Gb in size. I deleted all the files in .kde/share/apps/nepomuk/repository/main/data/virtousobackend/ directory and forced Nepomuk to re-index all my files and mails. This took 10 straigh hours. The PC was barely usable duing this time. But after the Nepomuk finished indexing the virtuoso-t was not taking any more CPU, only when automatic file indexing was updating the index. So, I ended up switching off the file indexing (suspending it) and only leaving mail indexing.

So, my suggestion: delete the Nepomuk index database from 4.7 and force it to recreate the index database and after this disable file indexing and leave at least 512Mb for virtuoso-t to have enough memory to do all the indexing in RAM, instead of using HDD. If you have less than 4Gb RAM, set it to less than 512Mb.

One more suggestion (from Alien Bob, the Slackware developer): when you upgrade the KDE always delete the KDE cache directories. The leftovers are always causing problems. Afer I upgraded I first logged in as root and deleted these 2 directories (which are symlinked in .kde directory):

.kde/~cache-username
.kde/~tmp-username

These are symlinks to /var/tmp/kdecache-username and /tmp/kde-username.

Try to delete these. It should help to avoid lot of troubles.

Can you check to see if this Virtuoso CPU usage is caused by Akonadi indexing (although you said it was running ok, so that shouldn't be the case) or if it is something else causing this?

You can view the state of Akonadi indexing using "akonadiconsole".

szob wrote:So, my suggestion: delete the Nepomuk index database from 4.7 and force it to recreate the index database and after this disable file indexing and leave at least 512Mb for virtuoso-t to have enough memory to do all the indexing in RAM, instead of using HDD. If you have less than 4Gb RAM, set it to less than 512Mb.

One more suggestion (from Alien Bob, the Slackware developer): when you upgrade the KDE always delete the KDE cache directories. The leftovers are always causing problems. Afer I upgraded I first logged in as root and deleted these 2 directories (which are symlinked in .kde directory):

.kde/~cache-username
.kde/~tmp-username

These are symlinks to /var/tmp/kdecache-username and /tmp/kde-username.

Try to delete these. It should help to avoid lot of troubles.

Thank you a lot, your suggestion helped. I have deleted tmp dirs, removed Nepomuk database, enabled only mail index and now it seems to work. There was bigger load only short time after KDE start and now it is OK. I hope it will stay in this state.

KejPi wrote:Thank you a lot, your suggestion helped. I have deleted tmp dirs, removed Nepomuk database, enabled only mail index and now it seems to work. There was bigger load only short time after KDE start and now it is OK. I hope it will stay in this state.

Glad it helped. I think the main problem is Nepomuk file indexing. This is what cause CPU overload.

How ironic that the Sematic Desktop developers gave the name of Nepomuk to this software. Probably, in Czech it means the same it does in Polish: he did not help (pronounced "nie pomugl", which sounds nearly the same as Nepomuk). This software surely does not help so much and its name reflects it. In fact, it makes things much worse for an average user.

EDIT: remember also about this (which was mentioned at the beginning of this thread:

High CPU or Memory usage
If you are experiencing 100% CPU usage by the virtuoso-t process when using Akonadi and related applications, try this proposed workaround while it is being investigated: In KRunner's configuration page, disable the Nepomuk search plugin and the Contact plugin. Then, log out and back in. For further information and inputs, report back here or on the Forum or on the IRC channel #kontact. (KDE userbase website).

I did disable these 2 items in KRunner, but CPU usage was still high by virtuoso-t. Only when I disabled the Nepomuk file indexing things settled down.

bcooksley wrote:Can you check to see if this Virtuoso CPU usage is caused by Akonadi indexing (although you said it was running ok, so that shouldn't be the case) or if it is something else causing this?

Akonadi was a problem in KDE4.7.4, since it had a lot of seg faults, which caused the e-mails to multiply after each crash. This was the reason I moved to 4.8.0. In the new KDE release akonadi is working OK, except akonadi_mixedmaildir_resource eats up 1.6Gb of RAM when I go through my mail folders and KMail does the folder synchronization, especially the ones with 20,000 or more mails.

If I stick to smaller folders, the memory usage is Ok. In my opinion akonadi should release this memory once I close KMail, but it is not the case. I need to either kill the process or re-log to KDE. So, I ended up switching off auto synchronization of the mail folder and only allowed synchronization of a folder when I highlight it. This is why I avoid the big folders and stick to the smaller ones with current mail. This is a bug for me and should be looked at.

bcooksley wrote:You can view the state of Akonadi indexing using "akonadiconsole".

Thanks. I checked it. Nice tool. I try to play around with Akonadi settings to see if I can get the memory problem solved.

so after a lot of reading and trying it seems either running "nepomukcleaner" or deleting the nepomuk database would resolve this problem.