This forum has been archived. All content is frozen. Please use KDE Discuss instead.

[KIO] Bigger buffers maybe?

4

Votes
4
0
Tags: None
(comma "," separated)
rt11
Registered Member
Posts
2
Karma
0

[KIO] Bigger buffers maybe?

Mon Feb 19, 2018 12:32 pm
So I'm totally new to this but I've been having a problem with NFS mounts and Dolphin. I've also sort of solved it, so I'm wondering if the solution is a good one that should be included in KDE going forward. My background is mostly Windows development, so I'm reluctant to start setting up a whole dev environment and stuff. I just want this one thing fixed. :)

Anyway, mount.nfs works in one of two very distinct modes: sync and async, the latter being the default. Sync mode behaves as you'd expect, but async mode is a little surprising. It's explained in detail here and also in man nfs, but the short version is that, when writing to a share mounted in async mode, no data is ever sent to the server until the file being written to is closed, or until the client runs out of dirty pages. In either case the calling process is blocked while the cache is flushing.

Now, since it's flushed over a network connection, and since the write cache can be huge (default vm.dirty_background_ratio=10 equates to 3.2 GB of write cache in my case), copying a large file to an NFS mount using e.g. Dolphin will lock up the application for up to 30 seconds on a GbE LAN connection.

What's more, the progress indicator becomes useless in this scenario. From the application's perspective data is being written to the destination file as fast as it can be read from the source file, but it's only when the progress bar reaches 100% that any writing actually starts. So the bar just races to 100% and sits there while Dolphin (or whatever file manager) is all frozen up.

Now, easy solution: add "sync" to your mount options. Now everything runs smoothly. But it also runs slower because the client has to wait for the server to ack every chunk of data before sending the next. In my case this overhead reduces the speed to about 40 MB/s over GbE, which is less of a dealbreaker than applications freezing for many seconds at a time but still disappointingly slow.

There is a simple fix to KIO that makes the problem go away, though: ioslaves/file/file.cpp and ioslaves/file/file_unix.cpp use a hard-coded buffer size of 32 kB for copying files. Changing that to 2 MB increases the copy speed in sync mode to 108 MB/s, more or less maxing out the connection.

So I made the change, compiled KIO and copied the new file.so into my libs and now everything is awesome. I'm a little worried because the buffer size is labeled MAX_IPC_SIZE hinting that there might be some reason why it's hard-coded to be so tiny, but I just can't see what it is and I haven't experienced any problems yet.

So, any thoughts on this? Is there a reason for the buffer size? Should it be a user setting, perhaps?
User avatar
pappl
Registered Member
Posts
22
Karma
0

Re: [KIO] Bigger buffers maybe?

Mon Apr 08, 2019 7:07 pm
One year passed and i have exactly the same issue still.
NFS file copy and Dolphin = Progress bar freezes until copy job finished.
:(


Bookmarks



Who is online

Registered users: bartoloni, Bing [Bot], Evergrowing, Google [Bot]