This forum has been archived. All content is frozen. Please use KDE Discuss instead.

KMail directory structure starts vanishing into default tree

Tags: None
(comma "," separated)
User avatar
lukasz_r
Registered Member
Posts
8
Karma
0
OS
I've been using Akonadi and Kontact since 2012. I've had 9 email accounts connected to KMail via POP3. Over the years I've created quite a rich email directory structure with thousands of emails.

I've been experiencing some problems with Akonadi from time to time, but a recent trouble has put me in a state of despair, no exaggeration.

The troubles started when my hard drive got full. I've got LVM thinly provisioned volumes and this unfortunately might happen. It actually happened to me once before, leaving Akonadi not operational, but I did have a backup from a few hours earlier, which solved the issue back then.

This time I've got a backup from when I tried to start Akonadi and from a few weeks ago. As I've said, Akonadi server became not operational after a reboot when my hard drive got full.

I made a full backup of this faulty state, i.e.
~/{.local,.config,.kde,.cache}
etc. directories.

This is a log of what happens when starting Akonadi from a faulty state:

Dec 08 19:20:51 ll kontact[5263]: org.kde.pim.kmail: "Unknown error."
Dec 08 19:20:51 ll kontact[5263]: org.kde.pim.kmail: "Unknown error."
Dec 08 19:20:51 ll kontact[5263]: org.kde.pim.messagelist: Failed to load tags "Unknown error."
Dec 08 19:20:51 ll kontact[5263]: org.kde.pim.akonadicore: Job error: "Unknown error." for collection: QVector()
Dec 08 19:20:51 ll kontact[5263]: org.kde.pim.akonadicore: Job error: "Unknown error." for collection: QVector()
Dec 08 19:20:51 ll akonadi_control[5299]: org.kde.pim.akonadicontrol: Application 'akonadiserver' exited normally...
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: Failed to remove runtime connection config file
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: process error: "Unknown error"
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: exit code: 1
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: stderr: "2019-12-08 19:20:46 139817029634240 [Note] /usr/libexec/mysqld (mysqld 10.2.19-MariaDB) starting
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: stdout: ""
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: arguments: ("--defaults-file=/home/luke/.local/share/akonadi/mysql.conf", "--datadir=/home/luke/.local/sh
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: executable: "/usr/libexec/mysqld"
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: Database process exited unexpectedly during initial connection!
Dec 08 19:20:51 ll akonadiserver[5306]: org.kde.pim.akonadiserver: database server stopped unexpectedly


To restore the faulty state, I've written a small script syncing files from a faulty state backup:

Code: Select all
#!/bin/bash

src_top_dir=my_external_drive_dir_with_a_backup
dst_top_dir=$HOME

rsync_args () {
   rsync -aPvhHz --delete "$@"
}

rsync_args "$src_top_dir/.config"/{akonadi*,kontact*,kaddressbookrc,email*,foldermailarchiverc,kmail*,mailtransports,specialmailcollectionsrc,templatesconfigurationrc} "$dst_top_dir/.config"
rsync_args "$src_top_dir/.local/share"/{akonadi_migration_agent,emailidentities*,kmail*,kontact*,local-mail*,kxmlgui5*,kaddressbook*,contact*} "$dst_top_dir/.local/share"
rsync_args "$src_top_dir/.kde/share/config/mailtransports" "$dst_top_dir/.kde/share/config"

rsync_args "$src_top_dir/.local/share/akonadi" "$dst_top_dir/.local/share"


Using the script I can always go back to the above mentioned faulty state and try some things. When I delete the Akonadi database, I can easily turn off syncing
~/.local/share/akonadi
by commenting a last line.

I've basically tried to restore my emails (actually Akonadi/KMail email visibility, because emails are there in the 8.7 GiB
~/.local/share/local-mail
directory) in three ways.

  • Remove
    ~/.local/share/akonadi
    and sync the files via the above script, without
    ~/.local/share/akonadi
    .

    Result:

    All email identities and accounts set properly, but with random (i.e. pointing to some deep subdirectory in my directory structure) "sent-mail" etc. directories instead of the original ones. The same goes for the filter rules: they are there with random directories set, instead of good ones. But "Local Folders" are set with a proper path to the maildir, `~/.local/share/local-mail`.

    I can see all my directories in the left KMail panel, but shortly after they start __disappearing__, one by one. Then a default directory structure (inbox, outbox, etc.) is created.

    I can also trigger left-pane directories to vanish by restarting "Local Folders". Here are the logs of this event:

    akonadictl log:

    org.kde.pim.akonadicontrol: Restarting application '/usr/bin/akonadi_maildir_resource'.
    QObject::killTimer: Timers cannot be stopped from another thread
    org.kde.pim.akonadi_indexer_agent: Xapian error in indexer: Document 388 not found
    org.kde.pim.akonadi_indexer_agent: Xapian error in indexer: Document 389 not found


    journal log:

    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 47
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 268
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 48
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 49
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 51
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 52
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 50
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 4
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 269
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 53
    Dec 08 19:33:02 ll kontact[6165]: org.kde.pim.akonadicore: not referenced 5
    (...) lots of messages with the numbers


    The vanishing of directories seems to be related to a creation of the `~/.local/share/.local-mail.directory` dot directory, where the directory structure is kept. But this directory has never been there in my installation. Akonadi completely ignores the `~/.local/share/local-mail` structure.

    I can keep the directory structure in a left panel a bit longer by updating the folders, then setting appropriate directories in account and filter settings, but at some random point in time they start disappearing anyway, which keeps me from turning on mail check, for fear of ending up with two email directories. Interestingly, restarting "Local Folders" doesn't make the directories disappear.
  • Same as first scenario, but after the left-pane directories were replaced by a standard directory structure, move away the
    ~/.local/share/local-mail
    directory and import that directory from a new location with the Import Tool.

    Result:

    The original directory structure but with `new` subdirectories. The problem is that the imported emails go into the Akonadi database, not into the
    ~/.local/share/local-mail
    directory.
  • Same as first scenario, but after the standard directories replaced my directory structure, add a local maildir pointing to my directory structure.

    Result:
    It seems I've got my directory structure, but I'm not sure if all emails are there.

Ok, that's all I've tried so far.

I appreciate your comments on how to make Akonadi not replace my own directories with its own default directories. It's really important to me, before solving the issue I won't check my email via KMail.

Thanks a lot for your answers!
User avatar
lukasz_r
Registered Member
Posts
8
Karma
0
OS
I've just seen the problem again, with offline (I don't do online when checking Kontact so that no new mails mess up with already messed up tree) Kontact.

The directory structure stayed OK for a day (after restoring from the backup and deleting
~/.local/share/akonadi
I run
Code: Select all
akonadictl restart
akonadictl fsck

a few times, had all the account and filter directories set correctly).

When I opened the email identities pane, the directories started vanishing again, replace with a default directory tree, and the
~/.local/share/.local-mail.directory
directory was created again.

The log of the event:
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 293
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 139
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 138
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 137
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 136
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 292
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 135
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 134
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 291
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 378
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 133
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 290
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 289
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 377
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 376
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 288
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 287
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 132
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 131
Dec 09 20:45:09 ll kontact[8603]: org.kde.pim.akonadicore: not referenced 130
(...)
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
(...)
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_mailfilter_agent[8758]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_mailfilter_agent[8758]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_mailfilter_agent[8758]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_mailfilter_agent[8758]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_mailfilter_agent[8758]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_mailfilter_agent[8758]: "Item query returned empty result set"
Dec 09 20:45:02 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_archivemail_agent[8750]: "Item query returned empty result set"
Dec 09 20:45:02 ll akonadi_mailfilter_agent[8758]: "Item query returned empty result set"
(...)
Dec 09 20:45:01 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:01 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:01 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:01 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
Dec 09 20:45:00 ll kontact[8603]: "Item query returned empty result set"
User avatar
raphaelra
Registered Member
Posts
209
Karma
0
OS
There is also another thread with similar problem ...

Did you activated archive agent ?
User avatar
lukasz_r
Registered Member
Posts
8
Karma
0
OS
raphaelra wrote:Did you activated archive agent ?


No, the archive agent is off.

So far I've inspected the logs from the faulty state and it turns out they complained about the
~/.local/share/akonadi/db_data/tc.log

file. Removing the file made akonadi start and I could see all the folders there, with the proper settings for accounts and filters. Yet the joy was too early since the folders started disappearing again!

What I did now was to pull a backup of that file from an earlier backup and try starting, stopping, fsck-ing and vacuuming akonadi. The folders are there with the proper settings, but the results of
Code: Select all
akonadictl fsck

depend on whether Kontact is running or not. If it's running, it seems OK, but when not, there are some scary results which were there when the folders were starting to vanish.

I'm slowly getting to the point of letting Kontact download the new emails and see what happens, yet I've developed what I can call "the Akonadi syndrome" - excessive fear of seeing folders vanishing. But I must finally do it.
User avatar
lukasz_r
Registered Member
Posts
8
Karma
0
OS
Well, the problem seems to be resolved, I've been using Akonadi for a month or so after the following trick:

  • restore all the files (see previous posts) from a post-crash backup
  • pull the
    /home/luke/.local/share/akonadi/db_data/tc.log

    file from a pre-crash backup

Interestingly, deleting the above file leaves Akonadi working for a moment, but the folders start vanishing anyway. The file needs to be pulled from a working Akonadi state backup.

Now I hope the dreadful error never comes back. Since I've got a spare machine at work, I'll see what happens when importing my mail tree into the newest Kontact suite, under Tumbleweed, which I'm really eager to try anyway.


Bookmarks



Who is online

Registered users: Bing [Bot], daret, Google [Bot], Sogou [Bot]