Try using "akonadiconsole": there look for "Nepomuk Email Feeder" and similar things, remove them, then add them back.

einar wrote:Try using "akonadiconsole": there look for "Nepomuk Email Feeder" and similar things, remove them, then add them back.

Thanks! That looks to be part of the problem, it was stopped because it thought the system was too busy. I removed and restarted it and it ran for a while and stopped again for the same reason. My system wasn't at all busy when chose to stop the second time - can the test for that be disabled or adjusted?

On edit: akonadi_nepomuk_feeder seems go into "quiet" mode after only a short run. It then gives the status message "System busy, indexing suspended" but appears to be running in a very slow mode, all this when the system is essentially idle. I can't tell that stopping and restarting gets me any further into the messages that have not yet been indexed but I believe that it does not. So still looking for a solution, maybe just a more specific one this time: how do I get akonadi_nepomuk_feeder to run flat out until it finishes?

The indexing is throttled from 4.8.1 onwards, it's never really "suspended", but just slowed down to prevent system slowness. As I told you, try *removing* it and adding it back. I suspect something broke and this may be a way to get rid of the issue.

einar wrote:The indexing is throttled from 4.8.1 onwards, it's never really "suspended", but just slowed down to prevent system slowness. As I told you, try *removing* it and adding it back. I suspect something broke and this may be a way to get rid of the issue.

I have repeatedly removed akonadi_nepomuk_feeder and added it back, I'm not sure of the count but well more than a dozen times. It appears to run for a while then drops into the slow mode (is it possible that it's not detecting "busy" properly? I have a 6 core processor and it never uses more than one of them). It seems to always start at the beginning of the emails, reprocessing the same messages until (I hope) it gets to the messages that are not yet indexed. So far as I can tell from the results of a search it never gets to the unindexed messages, at least the search for a work in the text of the message never brings the message up..

I do see a lot of these thrown to .xsession-errors, apparently from old/bad emails:

[/usr/bin/nepomukservicestub] "/usr/bin/nepomukservicestub(6244)" Soprano: "Invalid argument (1)": "http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#messageHeader has a max cardinality of 1. Provided 2 values - _:sc, _:rc. Existing - "

Seems to be about 6700 of them in .xsession-errors I cleaned out yesterday morning. I suspect that many are duplicates from re-encountering the same emails after a restart.

Any ideas on what it's choking on?

Those "cardinality" errors should have been fixed already, but I don't know if just in master or in the 4.8 branch. As far as I can tell, they're part of the problem, but likely not all of it.
Also as I said, the message with "indexing suspended" is a red herring, as the indexer does not suspend anymore since 4.8.1. It merely throttles the indexing so it doesn't consume too much CPU.

einar wrote:Those "cardinality" errors should have been fixed already, but I don't know if just in master or in the 4.8 branch. As far as I can tell, they're part of the problem, but likely not all of it.

$ kmail -v
Qt: 4.8.1
KDE Development Platform: 4.8.2 (4.8.2)
KMail: 4.8.2

Also as I said, the message with "indexing suspended" is a red herring, as the indexer does not suspend anymore since 4.8.1. It merely throttles the indexing so it doesn't consume too much CPU.

I believe you, the problem I have is that it doesn't use *enough* cpu on my machine - I'd rather it actually finished faster. It seems to throttle down very early on my 6 core processor, and to not throttle back up. But after several restarts of the feeder it says that indexing is complete, and the search still doesn't find the old emails.

Searching for "backplate" by subject brings up 13 emails, searching by "body of message" brings up one (should be 113), searching by "any header" brings up the original 13. The .xsession-errors for the "body of message" search seems to be this:

Code: Select all

"SPARQL"
"select distinct ?r ?reqProp1 (bif:concat(bif:search_excerpt(bif:vector('backplate'), ?v2),bif:search_excerpt(bif:vector('backplate'), ?v4))) as ?_n_f_t_m_ex_ where { { ?r <http://akonadi-project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/8abc0596-0085-401d-b4da-40fa3a232c86> . { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v2 . FILTER(bif:contains(?v2, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> ?v3 . ?v3 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v4 . FILTER(bif:contains(?v4, "'backplate'")) . } . ?r a <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#Email> . } . ?r <http://www.semanticdesktop.org/ontologies/2007/08/15/nao#userVisible> ?v1 . FILTER(?v1>0) . }"
void Nepomuk::Query::QueryServiceClient::close()
AgentManager::removeSearch 105
AgentManager::addSearch "select distinct ?r ?reqProp1 (bif:concat(bif:search_excerpt(bif:vector('backplate'), ?v2),bif:search_excerpt(bif:vector('backplate'), ?v4))) as ?_n_f_t_m_ex_ where { { ?r <http://akonadi-project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/8abc0596-0085-401d-b4da-40fa3a232c86> . { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v2 . FILTER(bif:contains(?v2, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> ?v3 . ?v3 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v4 . FILTER(bif:contains(?v4, "'backplate'")) . } . ?r a <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#Email> . } . ?r <http://www.semanticdesktop.org/ontologies/2007/08/15/nao#userVisible> ?v1 . FILTER(?v1>0) . }" "SPARQL" 105
[/usr/bin/nepomukservicestub] virtual void Soprano::Server::LocalServer::incomingConnection(quintptr)
void Soprano::Server::ServerCorePrivate::addConnection(Soprano::Server::ServerConnection*) New connection. New count: 12
[/usr/bin/nepomukservicestub] Soprano::ODBC::Connection::Connection() Soprano::Server::ServerConnection(0x7fbcfc180730)
Known subscriber "kontact" subscribes again
Database "akonadi" opened using driver "QMYSQL"
void Nepomuk::Query::QueryServiceClient::close()
void Nepomuk::Query::QueryServiceClient::close()
plasma-desktop(6154)/plasma StatusNotifierItemSource::refreshCallback: DBusMenu disabled for this application
[/usr/bin/nepomukservicestub] virtual void Soprano::Server::ServerConnection::run() thread done.
virtual Soprano::ODBC::Connection::~Connection() Soprano::Server::ServerConnection(0x7fbcfc180730)
[/usr/bin/nepomukservicestub] void Soprano::Server::ServerCore::serverConnectionFinished()
virtual Soprano::Server::ServerConnection::~ServerConnection() Removing connection
[/usr/bin/nepomukservicestub] void Soprano::Server::ServerCore::serverConnectionFinished() Connection removed. Current count: 11

The search using "complete message" gives these messages in .xsession-errors, I'm giving this because it's reporting an error:

Code: Select all

"SPARQL"
"select distinct ?r ?reqProp1 (bif:concat(bif:search_excerpt(bif:vector('backplate'), ?v7),bif:search_excerpt(bif:vector('backplate'), ?v8),bif:search_excerpt(bif:vector('backplate'), ?v10),bif:search_excerpt(bif:vector('backplate'), ?v6))) as ?_n_f_t_m_ex_ where { { ?r <http://akonadi-project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/8abc0596-0085-401d-b4da-40fa3a232c86> . { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#bcc> ?v2 . ?v2 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#cc> ?v3 . ?v3 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#to> ?v4 . ?v4 <http://www.semanticdesktop.org/ontologies/2007/03/22/nco#hasEmailAddress> ?v5 . ?v5 <http://www.semanticdesktop.org/ontologies/2007/03/22/nco#emailAddress> ?v6 . FILTER(bif:contains(?v6, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#messageSubject> ?v7 . FILTER(bif:contains(?v7, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v8 . FILTER(bif:contains(?v8, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> ?v9 . ?v9 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v10 . FILTER(bif:contains(?v10, "'backplate'")) . } . ?r a <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#Email> . } . ?r <http://www.semanticdesktop.org/ontologies/2007/08/15/nao#userVisible> ?v1 . FILTER(?v1>0) . }"
void Nepomuk::Query::QueryServiceClient::close()
AgentManager::removeSearch 105
AgentManager::addSearch "select distinct ?r ?reqProp1 (bif:concat(bif:search_excerpt(bif:vector('backplate'), ?v7),bif:search_excerpt(bif:vector('backplate'), ?v8),bif:search_excerpt(bif:vector('backplate'), ?v10),bif:search_excerpt(bif:vector('backplate'), ?v6))) as ?_n_f_t_m_ex_ where { { ?r <http://akonadi-project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/8abc0596-0085-401d-b4da-40fa3a232c86> . { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#bcc> ?v2 . ?v2 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#cc> ?v3 . ?v3 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#to> ?v4 . ?v4 <http://www.semanticdesktop.org/ontologies/2007/03/22/nco#hasEmailAddress> ?v5 . ?v5 <http://www.semanticdesktop.org/ontologies/2007/03/22/nco#emailAddress> ?v6 . FILTER(bif:contains(?v6, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#messageSubject> ?v7 . FILTER(bif:contains(?v7, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v8 . FILTER(bif:contains(?v8, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> ?v9 . ?v9 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v10 . FILTER(bif:contains(?v10, "'backplate'")) . } . ?r a <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#Email> . } . ?r <http://www.semanticdesktop.org/ontologies/2007/08/15/nao#userVisible> ?v1 . FILTER(?v1>0) . }" "SPARQL" 105
Known subscriber "kontact" subscribes again
void Nepomuk::Query::QueryServiceClient::close()
Database "akonadi" opened using driver "QMYSQL"
AgentManager::removeSearch 105
AgentManager::addSearch "select distinct ?r ?reqProp1 (bif:concat(bif:search_excerpt(bif:vector('backplate'), ?v7),bif:search_excerpt(bif:vector('backplate'), ?v8),bif:search_excerpt(bif:vector('backplate'), ?v10),bif:search_excerpt(bif:vector('backplate'), ?v6))) as ?_n_f_t_m_ex_ where { { ?r <http://akonadi-project.org/ontologies/aneo#akonadiItemId> ?reqProp1 . ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> <nepomuk:/res/8abc0596-0085-401d-b4da-40fa3a232c86> . { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#bcc> ?v2 . ?v2 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#cc> ?v3 . ?v3 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#to> ?v4 . ?v4 <http://www.semanticdesktop.org/ontologies/2007/03/22/nco#hasEmailAddress> ?v5 . ?v5 <http://www.semanticdesktop.org/ontologies/2007/03/22/nco#emailAddress> ?v6 . FILTER(bif:contains(?v6, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#messageSubject> ?v7 . FILTER(bif:contains(?v7, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v8 . FILTER(bif:contains(?v8, "'backplate'")) . } UNION { ?r <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf> ?v9 . ?v9 <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#plainTextMessageContent> ?v10 . FILTER(bif:contains(?v10, "'backplate'")) . } . ?r a <http://www.semanticdesktop.org/ontologies/2007/03/22/nmo#Email> . } . ?r <http://www.semanticdesktop.org/ontologies/2007/08/15/nao#userVisible> ?v1 . FILTER(?v1>0) . }" "SPARQL" 105
void Nepomuk::Query::QueryServiceClient::close()
void Nepomuk::Query::QueryServiceClient::close()
Error during adding a record to table "CollectionPimItemRelation" "Duplicate entry '105-487505' for key 'PRIMARY' QMYSQL3: Unable to execute statement"
Item with id  487505  already in virtual collection with id  105
void Nepomuk::Query::QueryServiceClient::close()

Unfortunately I'm not sure on the exact nature of the problem as I'm not able to reproduce it on the development version I'm using.