Reply to topic

Off-loading processing to other hosts

2

Votes
2
0
kjetilk
Registered Member
Posts
10
Karma
0
All,

I've moved to only have fanless devices in my office. I love working in an environment without that kind of noise. However, that means I have relatively low-power devices too. I have quite a few boxes in the basement, I have a server rack, actually... 8) Obviously, the future will see ever more powerful devices that can be made fanless, but I'm pretty sure, my server rack will always be more powerful. Also, I have a gigabit Ethernet network between these devices, meaning that certain operations could have been made faster by letting another host in the LAN participate in the processing. I do this already a bit, both Digikam and Amarok uses a MySQL db on a server in the basement. It could do much more though.

My suggestion is therefore to implement some generic functionality that can help applications offload processing to a different host in the LAN. These hosts should be able to come and go, and it should be possible to configure them centrally, not on a per-application basis.

There are several ways this could be done. One possibility is to seek inspiration in DVD::Rip, which uses a cluster to transcode. This relies on the user having a NFS infrastructure defined, and SSH to distribute load. This is probably the simplest to implement, and it can work for relatively large volumes of distributed data, especially if the processing can occur on the NFS server, but it places a burden on the user to set up everything correctly.

Another possibility is to implement a custom protocol (or indeed, use MPI), so that very little configuration needs to be done on participating hosts. If all data needs to be shipped back and forth, it would probably handle less data though. Perhaps the multithreading framework could be extended?

The third thing that could be supported is to have all indexing processes and stuff, things that typically run in the background, run on a remote host. They might well be indexing a remote NFS host anyway.

I know it has been discussed having this kind of capabililty in kdenlive, but I can imagine that it could be useful for for example Digikam and Brasero (which could do the same thing DVD::Rip does), and others that occasionally have some heavy processing. It could also open more opportunities, like in office environments that could share processing power.

Oh, BTW, can you imagine a Beowulf cluster of KDE desktops? ;-)
airdrik
Registered Member
Posts
989
Karma
1
OS
Certainly an interesting idea. I don't know how far it will go though.
Most desktop apps are supposed to be pretty light-weight, and what kind of background processing goes on and what kind of effort would be required to send that off to other servers depends heavily on the application and the task being performed. I suspect the effort to implement this feature may be more effort than is worth (especially since most KDE desktop users only have one machine, or otherwise don't have servers readily available to use for desktop app background processing).


airdrik, proud to be a member of KDE forums since 2008-Dec.
kjetilk
Registered Member
Posts
10
Karma
0
airdrik wrote:Most desktop apps are supposed to be pretty light-weight, and what kind of background processing goes on and what kind of effort would be required to send that off to other servers depends heavily on the application and the task being performed.


Yeah, it might. I think that the most reasonable way to start this would to be that those applications that would benefit most for it should look into their common needs. For me, that'd be digikam and kdenlive.

airdrik wrote: I suspect the effort to implement this feature may be more effort than is worth (especially since most KDE desktop users only have one machine, or otherwise don't have servers readily available to use for desktop app background processing)


That's true, but my thought was that if you have a many computers, like in an office, you could have a single server providing computing power to many fanless desktops. I would have liked that, it would have made cubicles slightly more bearable...
airdrik
Registered Member
Posts
989
Karma
1
OS
When looking at specific applications, I can definitely see kdenlive benefiting (if it doesn't already have this functionality) as offloading the rendering to a render farm seems like it should be a standard feature for a video processing application which aspires to anything more than casual use.
Digikam could also benefit when doing bulk image processing operations on a decently-sized library of images. Of course a tech-savvy user could probably turn this into a couple of scripts (using e.g. imagemagick) which could be distributed and kicked off on the processing server/cluster in a few different ways. Having that built into the app would certainly simplify that for the user.


airdrik, proud to be a member of KDE forums since 2008-Dec.

 
Reply to topic

Bookmarks



Who is online

Registered users: 1user2, 1ykos, aureotoshi, Baidu [Spider], bartoloni, Bing [Bot], Exabot [Bot], funkybomber, gdrift, gfielding, Google [Bot], jacob-em, johssw, jSim, Sogou [Bot], strag, thugo, tosky, Yahoo [Bot]