freininghaus wrote:I investigated this a bit and found a problem in the code that looks easy to solve, so I hope that we can include this in KDE 4.10.1:

https://git.reviewboard.kde.org/r/108984/

After the folder has been loaded, either previews are being generated or, if previews are disabled, the MIME type of all files is determined asynchrously. At the moment, every time we get a new preview or a MIME type (which determines the icon shown for the file), we indeed trigger a full re-layouting of all items.

There are some more inefficiencies in that area, but this patch is at least a first step towards making Dolphin behave better

Wow, thanks for looking into solving that! I hope the patch is good and will be in the next version of KDE (it sounds like 4.10 hasn't been released yet so maybe that will have it).

MirceaKitsune wrote:Wow, thanks for looking into solving that!

You're welcome

MirceaKitsune wrote:I hope the patch is good and will be in the next version of KDE (it sounds like 4.10 hasn't been released yet so maybe that will have it).

KDE 4.10 has been released on February 6: http://dot.kde.org/2013/02/06/410-relea ... t-platform

But the patch will be in KDE 4.10.1, which is going to be released next month. But as I said, scrolling a view with many items may still be not quite perfect yet, I think there are more places in the code which could be made more efficient.

freininghaus wrote:KDE 4.10 has been released on February 6: http://dot.kde.org/2013/02/06/410-relea ... t-platform

But the patch will be in KDE 4.10.1, which is going to be released next month. But as I said, scrolling a view with many items may still be not quite perfect yet, I think there are more places in the code which could be made more efficient.

Didn't know 4.10 was already released. And ok, it's great if the fix will be in! Was hoping openSUSE 12.3 would have the fix, but if it's next month it will probably be released with 4.10.0 and might not include the update later on :/

freininghaus wrote:I investigated this a bit and found a problem in the code that looks easy to solve, so I hope that we can include this in KDE 4.10.1:

https://git.reviewboard.kde.org/r/108984/

After the folder has been loaded, either previews are being generated or, if previews are disabled, the MIME type of all files is determined asynchrously. At the moment, every time we get a new preview or a MIME type (which determines the icon shown for the file), we indeed trigger a full re-layouting of all items.

There are some more inefficiencies in that area, but this patch is at least a first step towards making Dolphin behave better

Thank you. Unfortunately, I don't see any loading speed/cpu usage difference with and w/o this patch, when entering big dirs. Where should we look for improvements?

Another thing - I/O. There is huge difference between Dolphin and qtfm (for example). What's Dolpin doing there? http://youtu.be/tzKOCbQL15Y

gedgon wrote:Thank you. Unfortunately, I don't see any loading speed/cpu usage difference with and w/o this patch, when entering big dirs. Where should we look for improvements?

What I did to find out what was making Dolphin slow for some time after a folder was opened (which is what this topic is about) is the following:

1. Disabled natural sorting (which is a major bottleneck that may use a lot of CPU cycles and freeze Dolphin for some time *before* the folder contents are shown). That would make it hard to profile anything that happens *after* the contents are shown.
2. Opened Dolphin in a folder with many items with Valgrind/Callgrind, and waited until it did not use the CPU any more.
3. Analysed the output with kcachegrind.

This showed that around 12% of the total CPU cycles (i.e., 12% of all cycles required while Dolphin was running) were spent in repeated re-layoutings just after the directory was opened, and I believe that these were at least one reason why scrolling is slow just after opening the directory. These are gone with the patch.

gedgon wrote:Another thing - I/O. There is huge difference between Dolphin and qtfm (for example). What's Dolpin doing there? http://youtu.be/tzKOCbQL15Y

I don't know. Dolphin itself does basically not do any I/O. Files are accessed by KIO::PreviewJobs (to generate previews), by the mime type determination code from kdelibs, and by Nepomuk/Strigi. We have little influence on that (well, obviously, one can disable previews and Nepomuk and tool tips and the information panel, but at least the mime type determination remains).

freininghaus wrote:[...]by the mime type determination code from kdelibs[...]

Thank you for quick response. Is there any chance that you could fill proper bug report against kdelibs? Mime type determination code is clearly horribly ineffective.

gedgon wrote:

freininghaus wrote:[...]by the mime type determination code from kdelibs[...]

Thank you for quick response. Is there any chance that you could fill proper bug report against kdelibs? Mime type determination code is clearly horribly ineffective.

Well, for a "proper" bug report (which means for me, "a bug report that is likely to lead to an improvement in the code"), a lot more analysis, debugging, and profiling has to be done, and I don't have the resources to do that, sorry.

freininghaus wrote:[...] and I don't have the resources to do that, sorry.

How about workaround then? Krusader doing this quite well, by determining mime type only for "visible" (in view window) files.

gedgon wrote:How about workaround then? Krusader doing this quite well, by determining mime type only for "visible" (in view window) files.

Interesting idea. At the moment, we determine mime types (which are required to decide what icon to show if previews are disabled) and create previews (if they are enabled) for all items - first for the visible ones according to their order in the view, and then for all others. I haven't implemented that myself, but I guess the reason was to make sure that, after the initial loading/mime type determination/preview generation is completed, you never have to wait for icons/previews to arrive when you scroll down. But I see that this might not be the wisest thing to do if there are really many files in the directory.

Another approach could be to only do the expensive mime type determination/preview generation for the visible items and those that you can reach quickly by scrolling up/down one or two pages or by pressing Home/End, to minimize the irritations that could happen when you scroll to a position in the view where there are no icons/previews yet.

This would require quite a bit of restructuring in the class KFileItemModelRolesUpdater that does all that - it's just easier to do it for all items than to do it just for a subset, and adjust that subset dynamically as soon as the view is scrolled. Moreover, we'd have to add a special case for "Sort by type", in which case all mime types are needed anyway (but not all previews), but it should be possible. I can't make any promises at the moment though. I would first try to investigate the actual reasons for the current approach (to see if there were more than just convenience) and then find some time to write the code, or find someone else who does it.

That would be great.

btw

freininghaus wrote:[..]we determine mime types (which are required to decide what icon to show if previews are disabled)

Is this necessary? How about initial/fast icons assigning by type/extensions for all files and then correcting wrong ones using real mime type determination in background with lowest priority (in the way like you described), but always before executing?

I haven't checked the actual code in question, but KDE does not have any understanding of "file extensions" - except that they are attached to file mimetypes.

I don't know if there is a way to skip the "magic" based mimetype lookup (which looks for structures in the file) - but if there is, then that is what would give you the intended results.

We use KFileItem::determineMimeType() to determine the mime type, which checks extension and content. AFAIK, there is no API in kdelibs that permits to use the extension only.

QMimeDatabase in Qt 5 does provide something like that:

http://doc-snapshot.qt-project.org/5.0/ ... hMode-enum

But I'm not quite sure if the possible benefit of using that (i.e., very fast first guess for a mime type) is really worth the effort. Two-stage mime type determination would require quite a bit of complexity in the code, and might also be irritating in situations where the files are sorted by type and jump around multiple times if the "guess by extension" result was wrong.

freininghaus wrote:But I'm not quite sure if the possible benefit of using that (i.e., very fast first guess for a mime type) is really worth the effort. Two-stage mime type determination would require quite a bit of complexity in the code [...]

Correct me if I'm wrong, but I think Krusader doing something like that. I never saw there "default/unknown" file icon and entering big dirs is quite snappy, fast and light for CPU/storage device.

freininghaus wrote:and might also be irritating in situations where the files are sorted by type and jump around multiple times

When sorted by "type" Krusader relies on full mime type determination all files in directory, and then it is slow.

gedgon wrote:

freininghaus wrote:But I'm not quite sure if the possible benefit of using that (i.e., very fast first guess for a mime type) is really worth the effort. Two-stage mime type determination would require quite a bit of complexity in the code [...]

Correct me if I'm wrong, but I think Krusader doing something like that. I never saw there "default/unknown" file icon and entering big dirs is quite snappy, fast and light for CPU/storage device.

I cannot say what exactly Krusader does, but I guess that it just tries to determine the mime types of the visible items first. Actually, there is some code in Dolphin that tries to do the same, but I had a closer look yesterday and found out after some debugging that it does not work as it should. The reason is that the class that determines the mime types gets notified about the files in the directory before the view tells it which items are visible. Therefore, it tries to determine the mime types of all files in random order for 200 ms, and then shows the view, which might contain some "unknown" icons for files whose type could not be determined in 200 ms. A very stupid bug, but I'm afraid that it's not like it could be fixed with just a few lines of code

An update that might be useful. I think the issue is a little less severe since updating to KDE 4.10 (from 4.9) some months ago. Although I'm not certain, it appears to take 5 seconds where it would previously take 8 or something like that. Natural sorting was turned off both then and now, so it doesn't influence this calculation.

It still takes very long however. Both when I open the folder in Dolphin or browse there via the file saving dialog (eg: You select "save image as" in Firefox and KDE open its window to let you browse there). Hoping in the future more can be done to improve this... that would be very nice.