Introduction

Inspired by the recent turmoil with the lyricswiki-script, and based on the fact that scripts can easily be updated without recompiling or reinstalling Amarok, I had the idea that an automated updating system for scripts would be a great thing. Especially, this would mean that new versions of scripts can be published in a matter of minutes, without the need to release a new version of Amarok (or to wait for such a release).
Discussing the idea in IRC it quickly turned out that (a) I wasn't the first to think of it and (b) to prevent any security risks, digital signing would be necessary. Of course, updates would only be hosted on a trusted server (amarok.kde.org I guess), but you have to assume that each and every webserver is hackable, and an automated update system that distributes malicious code is quite a nightmare.
I went ahead and created first a small proof-of-concept app and then a slightly larger standalone app that provides the functionality we need, based on KDE and Qt technology. The cryptographic stuff that's needed for the signatures is provided by the "Crypto++" library, which therefore becomes a new dependency for Amarok. I have not yet actually integrated the new code into Amarok, but hopefully that will mainly be a matter of copy-'n'-pasting now.

What works:

• generating key pairs (small command-line app)
• creating signatures for files (same small command-line app, the two can easily be split if deemed appropriate)
• copying all scripts from /usr/share/apps/amarok/scripts/ to ~/.kde(4)/share/apps/amarok/scripts/, which is of course necessary to make the scripts replacable without requiring root access
• determining the version number of the existing scripts
• checking a predefined location for updates
• deciding on the applicability of those updates
• downloading the updates
• verifying the signature of the updates
• installing the updates (if verification was successful)

That's pretty much all we need

What remains to be done is integrating all of that into Amarok, and here I need some help:

• At which point exactly is it most reasonable to perform the updating procedure? My guess is dialogs/ScriptManager::ScriptManager(QWidget* parent), or the beginning of dialogs/ScriptManager::findScripts(), but I'm not sure whether there's a better solution.
• With the changes incorporated, Amarok will need Crypto++ both for compiling and at run time. Crypto++ is available via the package managers of Gentoo and Ubuntu; I didn't check other distros but they should have it, too. How does one have to change the build system to integrate the new dependency? (In the simple Makefile I've used for testing, the required parameter is "-lcryptopp")
• The (probably) biggest remaining change is that so far, I execute the downloads of version, update and signature synchronously; I guess that will have to be changed to asynchronous operation. Unfortunately, I have no idea how the Qt event system that's required for this works. All I can say is "SLOT? wtf?"
  Please, could someone either explain to me how I can use asynchronous downloads, or (even better ) offer to take over implementation of that part?
  There's one small catch: The contents of the method ScriptManager::findScripts() should only be executed after the updating process, asynchronous or not, has finished. Otherwise an additional restart of Amarok would be necessary in order for the updated scripts to be used.
• Which base URL should be used for the update checks? amarok.kde.org/updates/scripts/ ?
• The public key that must be integrated into Amarok is around 400 bytes of base64 encoded data. Should I put it as a string directly into the source code, or is it better to have it as a separate file? If the latter, where? I don't think there's a reason not to put it into the source, because we don't intend to change it; and if we do want to change it, that can happen in the next Amarok version.

And here is the (planned) workflow for releasing an update:

1. Joe User in a spate of industriousness creates an updated version of greatscript/main.js and greatscript/logo.png
2. He sends them to Dev E. Loper for review.
3. Dev verifies that the new version works and doesn't contain malicious code.
4. Dev especially makes sure that in the first line of main.js, the NUMBER in "AmarokScriptVersion=NUMBER" got incremented. (It doesn't really matter by how much, but there must be an increment to prevent endless updating loops. Only integers are allowed, no point releases.) Dev memorizes the new version number.
5. Dev puts the updated files into an archive called main.tar.gz (important: without any enclosing folder!)
6. Using his private key, Dev signs the archive, generating main.tar.gz.signature.
7. Assuming the base URL for updates is http://amarok.kde.org/updates/scripts/, Dev now puts the files online:

Code: Select all

mv main.tar.gz http://amarok.kde.org/updates/scripts/greatscript/main.tar.gz
mv main.tar.gz.signature http://amarok.kde.org/updates/scripts/greatscript/signature (note the renaming!)

And yes, I know that you can't use mv exactly like that. But you understand what I mean, right?
8. Finally, Dev tells Amarok clients all over the world that an update is available, using the version number he remembers:

Code: Select all

echo "<version number>" > http://amarok.kde.org/updates/scripts/greatscript/version

9. After that, when somewhere on the internet an Amarok is started, it will notice the update, download, verify and install it.
10. Dev and Joe pat each other on the back for having cooperated so well, dancing around the room because Amarok is ro'king so much

A few remarks:

• The filenames "main.tar.gz", "signature" and "version" are hard-coded into the updater, so it's important that the files are named exactly like that.
• Each script must have a "main.js" anyway, and a new requirement is that in the first line of that file, "AmarokScriptVersion=NUMBER" must be present. Whitespace and other comments are allowed as well. The line might for instance look like this:

  Code: Select all

  // info for update system: AmarokScriptVersion = 3   Do not change unless you know what you're doing!

  Updating clients compare the number in the first line of the existing file $KDEHOME/share/apps/amarok/scripts/greatscript/main.js with the contents of http://amarok.kde.org/updates/scripts/greatscript/version when they decide whether to perform an update or not.
• To let the clients find the update, the directory with the script's name ("greatscript" in the example) must be identical on the server and in Amarok's script directory.
• If an evil hacker takes over the server amarok.kde.org and puts harddrive-wiping code into all of the scripts stored there (and increments all version numbers), the installed Amaroks in the world will download the updates, try to verify their signatures with the public key that's integrated into Amarok, that verification will fail, and no update will be performed. As long as the developers who have access to the private key keep this key secret, it is guaranteed that each update was in fact signed by a developer.
• If an evil hacker manages, in addition to taking over the server, to change the public keys that are part of Amarok installations worldwide to match a private key that he generated himself, he would then be able to abuse the updating system for his purposes. On the other hand, if he's already managed to change data on your hard drive, you're f*cked anyway...

And for all the naysayers: (OK, I don't really mean that, a bit of skepticism is quite healthy)
What about security?

Well, I'm not Bruce Schneier, but as a computer science student I've got enough background information to know what I'm talking about. And what I say is: As described/implemented so far, the updating system is secure. At least, it's what's considered secure by everyone at the moment. That means:
Digital signing is intended to protect against:

• compromised update servers
• compromised connections to the server ("man-in-the-middle" attacks, compromised proxies/routers, or simply transmission errors)

and it achieves this protection successfully. It won't and can't protect against:

• trusted devs having their computers hacked or their houses broken into, where the private key might be stolen
• trusted devs signing unstable/malicious/**** code because they didn't see the code was bad
• trusted devs turning over to the Dark Side(TM)
• changed public keys on users' computers
• brute force attacks; but unless you've got a supercomputer at your disposal and several years of time on your hands you don't really have a chance to find the private key
• a scientific breakthrough that renders current encryption schemes weak (extremely unlikely, but can't be ruled out)
• earthquakes, so-called terrorist attacks, and the grass getting all wet in the rain.

So, relax, these remaining risks are (a) very very small and (b) anyway completely unavoidable. If you don't trust the Amarok devs' good intentions and competence, don't use Amarok. In fact, it's best not to use a computer at all.

Any thoughts? Any help? Of course, I can immediately put my source code here, if anyone wants to look at it.

Generally, nice idea, but it's a bit blown up, isn't it.
The scripts coming with Amarok don't need to be updated, as they will be with the next release.
Scripts in $HOME are third-party scripts, which are under the user's responsibility.
They got installed through download from kde-apps.org or from within amarok( ,which uses the same directory on kde-apps.org) without any keypairs.
Increasing the version would be the responsibility of the script developer.

Remove the keyring feature.
Updates only for extra scripts.

Or why don't you write a script, which updates itself first and the user-installed ones afterwards. It can grab the version info from the package version, the .spec file or from an extra version file, if you want.

No matter which solution:
The script developers need some guidelines and they should be liable for including this functionality.
And I'm in doubt they would do that, because most scripts even don't have a version in their tarballs.

I think Amarok devs have enough to do with Amarok itself, without verifying the code of third-party scripts.

m0nk

jmrk: Please send this to the mailing list, the devs hardly have time to look at the forum, so if you want then to look at that, the mailing list is the right place.

Dieter Schroeder wrote:The scripts coming with Amarok don't need to be updated, as they will be with the next release.

Obviously you haven't been following the lyricwiki script over the past 6 weeks, otherwise you wouldn't say that Amarok's scripts don't need to be updated between releases. And my updating system is intended for them, *not* for 3rd party scripts that are hosted on kde-apps.org.
Which scripts come packaged with Amarok, and for which scripts updates will be supplied, will of course always be for the developers to decide.


@Mamarok: thanks for the advice, I will do that.

OK, the lyrics script...
But that has been fixed by a patch (for those still using it).
It would be more interesting to get these unversioned third-party scripts under control, I think.
The handful scripts delivered with Amarok are easy to maintain. This would be like shooting with canons upon sparrows.


m0nk

Well, it would make much more sense for the whole KDE anyway. So this thread should be moved. Looks like perfect for Brainstorm, but you will need to shorten your description. I guess that should not be that difficult

Dieter Schroeder wrote:OK, the lyrics script...
But that has been fixed by a patch (for those still using it).

Exactly, so far you had to apply patches manually; what I propose is more or less to automate that for all users. And if you do that, it has to be done in a secure way.
Users of the stable Amarok (and that would be most users, I think, because distros usually don't provide git or beta versions, and you have to be quite a geek to install them yourself, and even more of a geek to install patches from the bugtracker yourself) don't have working lyrics since 3rd August. Even users of 2.2-Beta1 didn't get lyrics with that, because the lyrics script contained in the Beta1 worked from August 4th till September 1st, but Beta1 was only released on September 4th and the latest patch (from Sept 2nd) isn't contained in it yet.

Dieter Schroeder wrote:It would be more interesting to get these unversioned third-party scripts under control, I think.

Maybe so, but that's not what I'm after. Anyway, my updater would support those scripts as well without any additional changes, if it is decided that updates for 3rd party scripts are distributed via the same server.

Dieter Schroeder wrote:The handful scripts delivered with Amarok are easy to maintain. This would be like shooting with canons upon sparrows.

I don't think so. It's only around 100 lines of code (and one external, distribution-provided library). Besides, you never know what the future brings - external websites always are a moving target, and Amarok might get more and more scripts over time. The updating system is lightweight enough that I think on the whole it would be a Good Thing to have it.

Mamarok wrote:Well, it would make much more sense for the whole KDE anyway.

That would indeed be cool; unfortunately, the whole of KDE doesn't consist of easily replaceable JavaScript files. Even most plasma widgets require compilation. You could use such an updater for backgrounds, I guess; but then again, how often does a background image need to be updated to a new version?
So, to make larger parts of your desktop (besides Amarok scripts) automatically updatable, you'd need a completely different and really much larger system. I believe the established distros' package managers do that job quite well.


Why am I meeting so much opposition? It's not like I'm trying to harm anyone, or to make anyone spend his time working on my ideas. I'm just trying to help improving my favourite audio player. I've already written the code and it already works, all that's left is to integrate it into Amarok and provide the necessary online resources to make it useful.

This is not opposition, don't understand us wrong, but you should be prepared to get some arguing, especially from the developers. This is quite normal, as you are facing a team of 15+ people who have their say about the code.

JFYI, this would have to wait anyway, as 2.2 is in feature freeze right now. Also, if you don't want it to be applied to the kde-apps.org scripts, this would be reduced to updating 2 or 3 scripts on a very rare basis, as the scripts inside of Amarok 2 are not that many.

Mamarok wrote:This is not opposition, don't understand us wrong, but you should be prepared to get some arguing,

Some arguing is OK, I was prepared for it; in fact I created this forum post with the intention to face it / get it out of the way. It's just that from the very beginning, I've faced nothing but skepticism, and that's getting pretty tiring. I would have hoped for someone to say "cool thing, let's have it!".

Mamarok wrote: especially from the developers. This is quite normal, as you are facing a team of 15+ people who have their say about the code.

That's one of the reasons why I haven't yet tried to integrate it into Amarok myself, instead asking (see above) what would be the best way to do it, so that (if possible) no one disagrees with the solution and the patch, once existent, doesn't get rejected.
Unfortunately, nobody has answered to that question yet, all I've been hearing is why we don't need an updater or why we can't have it in its current form or how it should be totally different

Mamarok wrote:JFYI, this would have to wait anyway, as 2.2 is in feature freeze right now.

That's too bad, because I guess it'll be some months till the next release, and depending on how the lyrics situation develops, updating scripts might be really helpful in the meantime... Well, bad timing can't be helped.

Mamarok wrote:Also, if you don't want it to be applied to the kde-apps.org scripts, this would be reduced to updating 2 or 3 scripts on a very rare basis, as the scripts inside of Amarok 2 are not that many.

I didn't say that I didn't *want* it to be applied to 3rd party scripts; I just said that those weren't what I had in mind when I wrote the updater. It's not for me to decide what scripts it will be used for once it's deployed; as I said before, it would work with any 3rd party script without requiring changes to its code. And yes, I'm aware that the integrated scripts hopefully won't need to be upgraded very often. In the past weeks, however, that was quite different.

jmrk wrote:That's one of the reasons why I haven't yet tried to integrate it into Amarok myself, instead asking (see above) what would be the best way to do it, so that (if possible) no one disagrees with the solution and the patch, once existent, doesn't get rejected.
Unfortunately, nobody has answered to that question yet, all I've been hearing is why we don't need an updater or why we can't have it in its current form or how it should be totally different

What about having some patience? You only wrote that mail today, it's Monday, everybody is working in their day jobs and busy on preparing a release, what do you expect?

And I very much hope you not just add it yourself, this should only been done by people who work on Amarok on a regular basis or if asekd so by the developers. But I guess you know by now how code suggestion works. We have a feature plan and developer sprints where we agree on stuff to be added, just adding code only because it has been suggested is simply not the way it works.

Also, expecting that everybody rushes in and thinks this to be an indispensable feature to have in the Amarok code immediately is a bit much, don't you think? It simply doesn't work that way, suggest, submit the code for testing and wait for the developers to agree, this can't be done in a few minutes.

Patience, young padawan

Mamarok wrote:What about having some patience? You only wrote that mail today, it's Monday, everybody is working in their day jobs and busy on preparing a release, what do you expect?

I didn't mean to make the impression of being impatient. I know everyone's busy and the world doesn't revolve around my contribution. I was merely remarking on the fact that skeptical comments were very quick to be voiced, whereas I haven't heard anything positive so far.

Mamarok wrote:Also, expecting that everybody rushes in and thinks this to be an indispensable feature to have in the Amarok code immediately is a bit much, don't you think?

I never said that. I said I would have liked to hear *someone* say something *encouraging*. But we really needn't discuss this, I'll just wait for more opinions to be expressed, whatever their stance is going to be.

Mamarok wrote:And I very much hope you not just add it yourself, this should only been done by people who work on Amarok on a regular basis or if asekd so by the developers. But I guess you know by now how code suggestion works.

Well, if I had a bit more of an idea about how to do it, I would very much like to integrate it myself, create a patch against the current git master, and publish that patch for review and submission. After all, I think the less a developer has to work himself on my contribution, the happier he is. Since I don't have writing access to the git repository, changing the official codebase myself has neven been an option. Of course, I can also stop at this point and hand over whatever code exists now to the devs.

Mamarok wrote:It simply doesn't work that way, suggest, submit the code for testing and wait for the developers to agree, this can't be done in a few minutes.

Exactly. I suggested the feature, in this very thread, at the same time announcing that a lot of the work is already done. And I asked for some help. Now I'm waiting for an answer, for months if I have to, then I'll (probably - depending on the answer I get) create a patch so that the code can actually be tested inside Amarok, then submit the patch for review, and finally wait again for its inclusion in Amarok.

Dieter Schroeder wrote:OK, the lyrics script...
It would be more interesting to get these unversioned third-party scripts under control, I think.

Versioning support for 3rd party scripts (including ways for automatic updating) is high on our TODO, but it's also quite difficult to do. We will implement it eventually, no doubt about that.

But I have to stress that these two issues are largely unrelated, though both are important.

markey wrote:We will implement it eventually, no doubt about that.

Well, thanks to a few hints I got on IRC mainly by lfranchi, my automatic updater is fully implemented and functional since a couple of days. I haven't pushed it to my public git clone yet; I thought I'd wait until after 2.2 is released.

As I said before, it can be used for any script, both those Amarok ships with and 3rd party additions. We can talk about the details and further enhancements (that will no doubt be necessary) when you have time to concentrate on it, i.e. after 2.2 is released.