Hi

Let me first describe what I intend to do:

I recently got this idea into my head, how BT should be suited to playback video files even during download. The download speeds I achieve easily would allow that. Doing some research I stumbled onto a paper describing all necessary changes to the the chunk selection procedure. That's when I decided trying to implement something similar in a ktorrent plugin myself and started digging through the ktorrent source code.

Now to my problem:

Since the ChunkSelector gets created by the Downloader, I thought about supplying a custom ChunkSelectorFactory to the constructor of Downloader. Then I found out, that the Downloader gets created by TorrentControl in its setupData method, which is called by its init method. And the init method gets called by TorrentCreator and the Core (which creates the TorrentCreator). So it seems to me, that it is only possible to supply a ChunkSelectorFactory by rewriting the Core class. I don't know, if I missed anything, but this looks like a flaw in the class design to me.

So that leaves switching the ChunkSelector after the construction of the Downloader. But since the ChunkSelector is private to the Downloader and the Downloader private to the TorrentControl, there's no way I can do that either.

Any ideas on how to get my own ChunkSelector into the Downloader?

EDIT: I started out with the stable source and then switched to the 4.0beta1 snapshot to check if anything related to my problem changed

The ChunkSelectorFactory was added upon request of the KGet people, which have an own Core implementation

So at the moment this is impossible to do from a plugin. However given the fact that I'm planning to add a streaming plugin, like you want to do, the api will need to change.

I have no objection to do these changes now, certainly if you would make the plugin. Saves me some work.

Yes, when the api allows plugins to supply their own ChunkSelector, I'll be working on a streaming plugin. So I guess I'll have to check out the svn sources

I'm gonna make the necessary changes tomorrow.

Will keep you informed.

OK there is a now a new method in the TorrentInterface class named setChunkSelector.

This allows you to set your own custom ChunkSelector, the passed object must be allocated on the heap, and the TorrentControl will take ownership of it. Passing a 0 pointer will trigger a reset to the default ChunkSelector.

Is there any progress made on this plugin? It would be a really nice feature :)

It has been quite a while now, but I am finally making progress.

This is actually my first KDE/Qt project, so I had some trouble with getting started, and then I have been on vacation for two weeks. Hopefully I'll have the first running version of my plugin soon.

It's kind of embarassing, but I only realized now, that using the new setChunkSelector method is a bit tricky. Since I don't see a straightforward way to provide ChunkManager, Downloader and PeerManager for the constructor of the ChunkSelector class, I cannot use the new method like I intended. For now I'll work around that issue by first providing my own ChunkSelectorFactory and then calling setChunkSelector(0).

Today I managed to compile the first version of my plugin, which actually modifies ktorrent's chunk-selection behaviour.

In order to get the plugin running, there are two patches, which need to be applied to libktorrent:
Since TorrentControl takes ownership of the ChunkSelectorFactory:

Code: Select all

--- libktorrent/src/torrent/torrentcontrol.cpp  (revision 1114734)
+++ libktorrent/src/torrent/torrentcontrol.cpp  (working copy)
@@ -1785,6 +1785,8 @@
 
        void TorrentControl::setChunkSelectorFactory(ChunkSelectorFactoryInterface* csfi)
        {
+               if (custom_selector_factory)
+                       delete custom_selector_factory;
                custom_selector_factory = csfi;
        }


And in order to make linking against libktorrent work:

Code: Select all

--- libktorrent/src/download/chunkdownload.h    (revision 1114734)
+++ libktorrent/src/download/chunkdownload.h    (working copy)
@@ -87,7 +87,7 @@
         *
         * This class handles the download of one Chunk.
        */
-       class ChunkDownload : public QObject,public ChunkDownloadInterface
+       class KTORRENT_EXPORT ChunkDownload : public QObject,public ChunkDownloadInterface
        {
                Q_OBJECT
        public:


Moreover after adding the code inside the archive to the plugin folder, the CMakeLists.txt file has to be adjusted:

Code: Select all

--- src/ktorrent/plugins/CMakeLists.txt (revision 1114002)
+++ src/ktorrent/plugins/CMakeLists.txt (working copy)
@@ -28,3 +28,4 @@
 macro_kt_plugin(ENABLE_SHUTDOWN_PLUGIN shutdown shutdown)
 macro_kt_plugin(ENABLE_ZEROCONF_PLUGIN zeroconf zeroconf)
 macro_kt_plugin(ENABLE_MAGNETGENERATOR_PLUGIN magnetgenerator magnetgenerator)
+macro_kt_plugin(ENABLE_STREAMING_PLUGIN streaming streaming)


There are still a few issues though.

• Since I haven't done much GUI coding before, I tried to keep it simple for now and added a entry to the context menu, like the downloadorder plugin does. Instead of a KAction I chose a KToggleAction and added my plugin as ViewListener to the TorrentActivityInterface. The plugin remembers the toggled state in a map and I tried to set it back when currentTorrentChanged is called. Unfortunately there is still something wrong with it, that I can't spot. Turning streaming on for one torrent and then repeatedly switching between two torrents (one with streaming, one without) and checking the state of the KToggleAction leads to erroneous behaviour. It alternately shows both of the torrents as streaming and non-streaming.
• A second issue concerns the algorithm used by the StreamingChunkSelector. Since I didn't implement a deadline, after which notyet-downloaded chunks are discarded and marked as missed (like the paper proposes), the high-priority section inside the download is constantly growing and hence the download slowly reverts back to most-rare-piece first. I have two ideas that could counter that effect. The first option is to slowly decrease the number of chunks inside the high priority section. This might still leave some of the lower chunks up to the finish of the download though. The second option is to raise the priority of the lowest chunks of the high priority section. The problem here is to decide when and how many chunks should get their priorities raised. I guess I'll probably try a combination of both unless I can think of something else.

Seth, I will try out your plugin probably tomorrow when I have some more time.

GUI wise it would make a lot of sense to put the actual GUI stuff in the mediaplayer plugin. You see ideally you want to be able to play a video while you are downloading with the mediaplayer plugin, and you want to be able to pause the video if the next chunk which needs to be played is not there yet.

So if you start playing an incomplete file you could then add your custom chunk selector, and then feed phonon the video data as it comes in. If the next part of the video isn't available, you could then do like youtube and stop playing until the data is there.

Yes I agree. In the long run it makes the most sense to integrate the new chunk selector into the mediaplayer plugin. Especially because that should make pausing the playback for missing pieces possible. I tried to keep the gui stuff minimal for now, just enough to test the actual algorithm, but somehow it got messed up anyhow.

Still, I'm far from being satisfied with the current state of my chunk selection algorithm (mainly because of the above mentioned issue). I don't have much time today, but I'll start to further improve it tomorrow.

Ok, so I adjusted the algorithm. It's working better now. I still couldn't figure out the gui issue though. And there's another issue:
The selection algorithm currently seems to waste some bandwidth. I'm guessing that happens, when multiple peers work on the same chunk? I already reduced the number of peers allowed to work on the same chunk, but still...

Also it would be really cool to take into account the average downloadspeed of the peer for choosing the chunks. That could prevent slow peers getting assigned early chunks and thereby blocking their download.

seth wrote:Ok, so I adjusted the algorithm. It's working better now. I still couldn't figure out the gui issue though. And there's another issue:

I think calling setChecked triggers the toggled signal, causing your plugin to enable or disable the streaming mode. By doing this, it doesn't seem to happen anymore:

Code: Select all

streaming_on->blockSignals(true);
streaming_on->setChecked(tc && i != streaming_torrents.end() && i->second != NULL);
streaming_on->blockSignals(false);


The selection algorithm currently seems to waste some bandwidth. I'm guessing that happens, when multiple peers work on the same chunk? I already reduced the number of peers allowed to work on the same chunk, but still...

Yes multiple peers will result in wasting bandwidth, the trouble is that you want to finish the chunk as fast as possible, but you don't know how fast a peer is going to upload it to you. So the current algorithm is to assign the same piece to multiple peers when more then one is assigned to a chunk. This is acceptable for the end of the torrent (the wasted bandwidth is the price to pay to get things downloaded faster).

However for streaming mode we need something better it will probably become to much.

Also it would be really cool to take into account the average downloadspeed of the peer for choosing the chunks. That could prevent slow peers getting assigned early chunks and thereby blocking their download.

We could keep track of the average speed since the last unchoke, and work with that. That would probably be a good means of choosing which peer to assign to what chunk. To get the best results, we probably need to assess the situation continuously and move the best peers around to the most critical chunks.

I will add some code to keep track of the average download speed since the last unchoke.

Has there been any further development on this plugin?

Hey guys, how's it going? Any noticeable achievements so far?