Hi all

I'm new to the development of KTorrent so I can use some little help. I'm working on a DHT manager to speed things up on local networks.

What I like to know is how the current DHT implementation works:

-how can i bypass the current handling of the DHT so that my 'system' fully manages the DHT? (where is it started?)

- if I start a torrent with a tracker... does KTorrent also use his DHT table? (my extension will manage the DHT always... trackerless or not)


I don't need very detailed information... just some pointers, hints or docs. Now I see only lots of files but I can't figure out how the work together... first time that I join a 'big' project like this.

Thanks

Bye


EDIT: is is no longer related to DHT

GrannySmith wrote:Hi all

I'm new to the development of KTorrent so I can use some little help. I'm working on a DHT manager to speed things up on local networks.

How do you intend to do that ? And why are you doing this ?

What I like to know is how the current DHT implementation works:

-how can i bypass the current handling of the DHT so that my 'system' fully manages the DHT? (where is it started?)

It is started or stopped in KTorrent::applySettings

- if I start a torrent with a tracker... does KTorrent also use his DHT table? (my extension will manage the DHT always... trackerless or not)

Yes, unless it is a private torrent.

I don't need very detailed information... just some pointers, hints or docs. Now I see only lots of files but I can't figure out how the work together... first time that I join a 'big' project like this.

Conceptually speaking the DHT consists of these things :
- A routing table containing all nodes, the table is used to decide which peers to contact
- RPC calls, which are requests to a node, and hopefully you will get back a reply
- Tasks : uses RPC calls to do announces and to find new nodes in the DHT network (the routing table is used to decide the initial nodes to contact)
- A database to store the lists of peers you find for each torrent

George wrote:How do you intend to do that ? And why are you doing this ?

How? I will use Avahi to do this (Avahi is a Zeroconf implementation). Why? Well... it is a school project. And it is interesting because it will save bandwith (and thus money if you have limits like me) and will speed things up on local networks.

George wrote:Yes, unless it is a private torrent.

Perfect

George wrote:Conceptually speaking the DHT consists of these things :
- A routing table containing all nodes, the table is used to decide which peers to contact
- RPC calls, which are requests to a node, and hopefully you will get back a reply
- Tasks : uses RPC calls to do announces and to find new nodes in the DHT network (the routing table is used to decide the initial nodes to contact)
- A database to store the lists of peers you find for each torrent

I think I will take a closer look at those tasks. If I can define some new tasks then my solution will almost perfectly blend within the current design/code.

Thank you for the feedback !!!

So you are trying to find DHT nodes on the local network by using zeroconf ?

Yes... that is the basic idea.

We've found out that downloads are controlled by a class called 'TorrentControl' AND by the DHT itself (both interfere with eachother). Is this correct?

I thought that there was only one control mechanism that took care of both parts by merging the information provided by those classes/parts. So when we include avahi then it wil existist of 3 parts... and they all are interfering with eachother.

I can not find something that simply says: startDownloadWithTracker(), stopDownloadWithTracker(), startDownloadWithDHT(), stopDownloadWithDHT().

(the above is ofcourse a simplistic view)

Anyway... I wondered why it is done this way... I can not find a reason why the TorrentControl class doesn't provide this kind of functionality. Or did I miss something important here (or in the bigger picture)?

At this moment we're working on a sort of avahi-wrapper that wraps the stuff we need in something that is easy to use. We already know that we can use a method to ping a 'node' if we provide an IP-address and a port number (method is part of the DHT subsystem).
But I've a little question about it.... when it pings does it gets information back of ALL of the files that the 'node' shares? If that is the case than we don't have to use avahi everytime :p

Thanks for the information and we keep you updated (if you're interested ofcourse)... if you have any additional information that could be usefull then feel free to share.

Bye

GrannySmith wrote:Yes... that is the basic idea.

We've found out that downloads are controlled by a class called 'TorrentControl' AND by the DHT itself (both interfere with eachother). Is this correct?

I thought that there was only one control mechanism that took care of both parts by merging the information provided by those classes/parts. So when we include avahi then it wil existist of 3 parts... and they all are interfering with eachother.

Downloads are only controlled by TorrentControl, the DHT is only used to find more peers, TorrentControl uses the DHT to start announce tasks (see PeerSourceManager and DHTTrackerBackend).

I can not find something that simply says: startDownloadWithTracker(), stopDownloadWithTracker(), startDownloadWithDHT(), stopDownloadWithDHT().

(the above is ofcourse a simplistic view)

Anyway... I wondered why it is done this way... I can not find a reason why the TorrentControl class doesn't provide this kind of functionality. Or did I miss something important here (or in the bigger picture)?

DHT can be enabled or disabled per torrent, this is done by startDHT and stopDHT in TorrentControl, so you did miss something. (It's a big class)

At this moment we're working on a sort of avahi-wrapper that wraps the stuff we need in something that is easy to use. We already know that we can use a method to ping a 'node' if we provide an IP-address and a port number (method is part of the DHT subsystem).
But I've a little question about it.... when it pings does it gets information back of ALL of the files that the 'node' shares? If that
is the case than we don't have to use avahi everytime :p

A ping is just to establish wether a node is alive.
The DHT does not have the functionality to see all the torrents a peer is sharing, the only purpose it has is to find peers for a torrent,

Thanks for the information and we keep you updated (if you're interested ofcourse)... if you have any additional information that could be usefull then feel free to share.

Bye

Sure, I'm interested, though I'm not really sure about DHT and zeroconf. I mean wouldn't it just be better to use zeroconf to see if there is anybody on the local network sharing the same torrents as you do ?

Isn't it more appropriate to use Zeroconf to implement an "Azureus LAN Peer Finder" type system, to locate those on the LAN downloading the same torrents as you, so that you can work together?

I think that'd be a great plus for KTorrent. It'd be even nicer if the LAN clients can cooperate in ways, like:
(1) Having LAN clients go by a different bandwidth limit setting, or bypass altogether.
(2) Prioritizing getting pieces from LAN peers over other peers
(3) Give more trust to LAN peers, such as a more generous choking algorithm

Wow... thanks for all the information... apparently we missed some major stuff (damn deadlines ).

With the new information about the *real structure* it sounds more reasenable to make an Avahi subsystem that is controlled by TorrentControl.

@jdong: Great suggestion... I'll check it out as soon as I submit this post.


*Goes back to the drawingtable*

Update: We can now find (using Avahi) and connect to local peers... it works great even without a tracker or dht.

So the next step is to optimize it by the following rules:

1. chunks available on the local network will be downloaded from the local network
2. download chunks on the local network first (optional)
3. ... (suggestions?)

Now I'm looking where KTorrent chooses which chunk will be downloaded next and from which peer. Can anybody give me a hint?

I already looked at the downloader, chunkdownloader, chunk, etc... but I can't find (or I missed) the spot where he makes the 'decision' (or random).

edit:
After some more research I've found out where chunks are selected but I can't find out where the peer is selected.
It looks like all peer are used... and the peers asks which chunks to download next... right?

ChunkSelector, we use a priority system and the rareness of chunks to select which one to download for a given peer.

OK... I think/hope that we are almost done...

we know now which peers are local and which are not local, we also changed the sort function for the chunks so that the local chunks are placed before the non-local chunks.

The only thing we miss is that we don't know which chunks are local, that is... were do you 'register' which chunks belongs to which peers.

We found the following function:

Code: Select all

void bt::Peer::haveChunk     (      Peer *       p,
      Uint32     index
   )     [signal]


but we're not sure that this is the right one... [signal]'s are new to us... never used it before.

Ok... we got it working.

It need some fine tuning/tweaks and more testing but the first set of tests passed perfectly. I hope to finnish it in the next 2-4 weeks.

GrannySmith wrote:OK... I think/hope that we are almost done...

we know now which peers are local and which are not local, we also changed the sort function for the chunks so that the local chunks are placed before the non-local chunks.

The only thing we miss is that we don't know which chunks are local, that is... were do you 'register' which chunks belongs to which peers.

We found the following function:

Code: Select all

void bt::Peer::haveChunk     (      Peer *       p,
      Uint32     index
   )     [signal]


but we're not sure that this is the right one... [signal]'s are new to us... never used it before.

I wouldn't rely on this signal, this is only for HAVE packets received from peers. A peer might allready have parts of the torrent when you connect to it.

We have a BitSet for each peer, which keeps track of which pieces a peer has, this should be used to see if a peer has a chunk.

Yeah... I used a combination of both (the signal and the bitsets) to make sure... I'll probably have to refine the current code and do a lot of test to make sure that everything is working properly.

I'll also check everyline of code again (actually it is not that much code... most of my time went to research and reading the docs and the code) and maybe I make it better/faster (sometimes it takes a while -1 minute- before changes are noticed)

I think that it is ready for beta testing

It is much faster now (almost instant) and it passes all our basic tests (1 local seed and 1 other local peer, a tracker and a lot of non-local peers)

I'll do some more tests (more local seeds, more local peers, ...) Friday or early next week... and when thats done I'll provide a patch.



Thanks again for all the hints !!!