While debugging some rtorrent stuff, I noticed that ktorrent (2.1.4 and svn) downloads much slower from a local(host) azureus tracker than rtorrent (and qbittorrent). File with a 512KB chunk downloads at constant ~2MB/s and file with a 128KB chuck at ~512KB/s (note the 4x rule), while in rtorrent/qbittorrent it goes 10+MB/s. Tried fiddling with that CPU Usage slider in svn, but it didn't help. Incidentally, just checked with transmission too - both torrents go at ~1,7MB/s.

Just letting you know.

That's not good, we should be flying with a local transfer.

I will check this out tomorrow.

OK, it seems that this has been the case ever since we added the changes to up the speed significantly (I managed to get around 6 MB/s over a LAN).

When I added those, I probably used a torrent with a big chunk size to test this.

OK, the 4 times the chunk size is no longer a problem, it seems we didn't assign new chunks fast enough.

Seems to work a lot better, although there's still something fishy. ~230MB/ (128KB chunk) file goes quickly to ~10MB/s and stays there, but the ~675MB (512KB chunk) file reaches ~10MB/s, but fluctuates between ~1-10MB/s. I'll do some more testing.

-edit-

Certainly something to do with chunk size still. A file with 4MB chunk doesn't want to go over 1MB/s.

Will check it out

I just tested a 4 MB chunk over a LAN, and it reached the speeds I was expecting (5 MB/s and more)

Not trusting ktorrent/rtorrent/azureus speed values, I just used a stopwatch. rtorrent downloaded a 676MB file (hosted locally in azureus) in ~50 seconds (meaning ~13.5MB/s average), ktorrent took twice as long (~100 seconds, ~6.75MB/s). rtorrent is designed to be very efficient though, could be what's making the difference. I'm not gonna cry over that difference, as I'm on a 1Mbit anyway

The fact is there's an improvement after fixing the issue with not assigning chunks fast enough.

RTorrent likes to cheat by downloading sequentially when it notices that the pieces have the same rarity. Need to fix this so that it changes starting point at regular intervals, except for that it is imo sane behavior.

So if you're getting just half the transfer speed, this might be a pointer though I'm not familiar with the KTorrent code.

Could the order be this important ?

Well, seeks do cost some 7-10ms, though there might be more than just the seeking due to piece boundaries.

I usually run tools like MacOSX's iostat to check how often my client does seeks. The problem with random access isn't just the seek, but also the read-ahead data that you tend to lose. Note that after reading in data, you also have to write it out so you're likely to end up seeking twice (at least) per piece.

For 6.75Mb/s with 512kb pieces, that might be more than 250ms lost to seeking.

Well, that could mean a possible improvement when downloading or uploading sequential chunks, but it seems to me that in most cases, the internet connection is so much slower than the how long it takes to access the hard drive, that you would have to have a _really_ good internet connection for it to make a difference - and even then I question it.

It certainly should improve the performance of your computer if KTorrent is set up to minimize disk accesses - or in this case the seek portion of the access at least - so it might be worth making KTorrent choose sequential chunks when chunks have the same rarity, but I don't think that it'll make much difference to how quickly a torrent gets downloaded.

Kalessin wrote:Well, that could mean a possible improvement when downloading or uploading sequential chunks, but it seems to me that in most cases, the internet connection is so much slower than the how long it takes to access the hard drive, that you would have to have a _really_ good internet connection for it to make a difference - and even then I question it.

It certainly should improve the performance of your computer if KTorrent is set up to minimize disk accesses - or in this case the seek portion of the access at least - so it might be worth making KTorrent choose sequential chunks when chunks have the same rarity, but I don't think that it'll make much difference to how quickly a torrent gets downloaded.

Obviously, it will not really matter much for most people, who can only dream of these internet speeds.

Rakshasa wrote:Well, seeks do cost some 7-10ms, though there might be more than just the seeking due to piece boundaries.

I usually run tools like MacOSX's iostat to check how often my client does seeks. The problem with random access isn't just the seek, but also the read-ahead data that you tend to lose. Note that after reading in data, you also have to write it out so you're likely to end up seeking twice (at least) per piece.

I don't know, we use mmap for most chunks (those who are in one file), I would expect that linux is a bit smart about this and keeps track of where the mmapped region of memory is located on disk, eliminating the need for a second seek.

Maybe we could make up for the seek somewhat with posix_fadvise, without having to add code to it all sequentially.

What I meant was that when you write a (random) set of pieces to disk, they are not likely to be continuous. And dirty pages get written within a certain time, often way before you gather enough to merge them much. The read I was talking about would be on the seeding side, since this test was run on one computer afaict.

And posix_fadvise gives limited gains as the default read-ahead is some 256K to 1M, which is pretty close to what you get with fadvise due to the size of pieces.