Reply to topic

Kdenlive doesn't max the CPU?

User avatar Ranko Kohime
Registered Member
Posts
41
Karma
0
OS

Kdenlive doesn't max the CPU?

Wed Jul 23, 2014 6:20 am
This is more of a question as to the why rather than a "how do I fix this?".

I recently built a new computer, with an Intel i7-4790k. So, ehm, yeah, 1080p videos now render about 200% of their playback speed. :D

But curiously, Kdenlive doesn't max out any of the cores, or the hyperthreads, according to any of the monitors I have available (GKrellM, KSysGuard, Htop), even when set to 8 threads in the render window. (And to be clear, yes the render window, not the settings window)

I ruled out disk throttling by placing the source and destination videos in a RAM fs, but it didn't seem to have any effect, and thermal throttling isn't the issue either, as the CPU could not exceed 60c during rendering, while during a Prime95 torture test it can get MUCH hotter.

Would anyone be able to shed light on this odd behavior so my curiosity can stop bugging me? ;D

For reference, I've also used an i3-2310m, and an AMD A8-6600k for rendering, (and the later a quad-core overclocked to 4.5GHz), and both of these Kdenlive was able to max-out.
User avatar TheDiveO
Registered Member
Posts
203
Karma
2

Re: Kdenlive doesn't max the CPU?

Wed Jul 23, 2014 5:05 pm
Same over here, on an i7 3.something GHz. I've once asked Dan Dennedy of MLT fame about this as this is about MLT multithreading and not Kdenlive. His answer was slightly disappointing (to me) and basically to the point that MLT can't max out thread usage anayway so there will always be some considerable amount of idleness. At least on more powerful systems with many cores. He doesn't plan on changing this situation, as all efforts are currently invested on movit instead and this is much more promising, especially for users with less powerful CPU hardware.

So, yes, we users with higher end i7 systems are somehow left in the cold as MLT cannot drive octocores at full load. I've noticed similar behavior to what you describe: my much lesser i5 quadcore gets maxed out, while the shiny octocore i7 won't. Damn, now we're not the coolest guys in town ... I'm afraid. xD


TheDiveO on YouTube: diving videos and aerial (quadcopter) videos, all made with Kdenlive.
TheDiveO Blog: know-how about Protune-grading in Kdenlive, GoPro HD Hero 3, DJI Phantom Quadcopter, full face mask diving, and some more...
kgolubev
Registered Member
Posts
19
Karma
0

Re: Kdenlive doesn't max the CPU?

Wed Jul 23, 2014 7:58 pm
I think I know why. Audio effects are tight to real time. So, it must play at normal speed. One thing you, can do - run two or more renderers :)
User avatar Ranko Kohime
Registered Member
Posts
41
Karma
0
OS

Re: Kdenlive doesn't max the CPU?

Thu Jul 24, 2014 9:49 pm
kgolubev wrote:I think I know why. Audio effects are tight to real time. So, it must play at normal speed. One thing you, can do - run two or more renderers :)

I would actually tend to doubt that. My i3 laptop had no trouble rendering, even though some of my 1/2-hour long videos were taking upwards of 6 hours to render, and on the other hand, even though my CPU isn't being maxed out, I'm still rendering in half the runtime of the video on the new rig.
User avatar Steve Guilford
Registered Member
Posts
151
Karma
0

Re: Kdenlive doesn't max the CPU?

Thu Jul 24, 2014 10:04 pm
Bear in mind Linux is a time-shared O/S - not a RTOS (e.g. QNX or RT-11...that's ancient !!!). Therefore, the OS tries to prevent applications from maxing out the CPU by scheduling jobs at regular intervals - whether they like it or not. A process can 'nice' it's way to a higher priority but it's still up to whims of the OS's scheduler as to when you get to play with the CPU. If you want performance then you'd use runlevel 3, eliminate unneeded processes and a deploy a headless rendering engine.
User avatar TheDiveO
Registered Member
Posts
203
Karma
2

Re: Kdenlive doesn't max the CPU?

Fri Jul 25, 2014 3:16 pm
Nope, as Dan Dennedy explained it is difficult to even distribute work among eight threads in MLT. This is not a RTOS or any other OS scheduling problem. This is an application work distribution problem inside the application. Please don't mix up completely separate things here, unless you are clearly joking. But then, the irony must have slipped past me...


TheDiveO on YouTube: diving videos and aerial (quadcopter) videos, all made with Kdenlive.
TheDiveO Blog: know-how about Protune-grading in Kdenlive, GoPro HD Hero 3, DJI Phantom Quadcopter, full face mask diving, and some more...
User avatar Ranko Kohime
Registered Member
Posts
41
Karma
0
OS

Re: Kdenlive doesn't max the CPU?

Mon Jul 28, 2014 8:58 pm
So, weird experiment produces weird results: I've been manually throttling my CPU when I plan to leave it alone for a while (while sleeping or out of the house), as for some reason automatic throttling isn't working in Arch. I started a render at the lowest performance level of 18%, (800MHz), then cranked it to 50% (2GHz), then cranked it to 100% (4GHz).

Melt continued to consume the same percentage of CPU time on average through these changes, and the rendering in Kdenlive sped up as a result. Weird!
User avatar Steve Guilford
Registered Member
Posts
151
Karma
0

Re: Kdenlive doesn't max the CPU?

Mon Jul 28, 2014 9:32 pm
Ranko Kohime wrote:So, weird experiment produces weird results: I've been manually throttling my CPU when I plan to leave it alone for a while (while sleeping or out of the house), as for some reason automatic throttling isn't working in Arch. I started a render at the lowest performance level of 18%, (800MHz), then cranked it to 50% (2GHz), then cranked it to 100% (4GHz).

Melt continued to consume the same percentage of CPU time on average through these changes, and the rendering in Kdenlive sped up as a result. Weird!

Not weird. That sounds normal.
User avatar Ranko Kohime
Registered Member
Posts
41
Karma
0
OS

Re: Kdenlive doesn't max the CPU?

Wed Jul 30, 2014 7:04 am
Steve Guilford wrote:
Ranko Kohime wrote:So, weird experiment produces weird results: I've been manually throttling my CPU when I plan to leave it alone for a while (while sleeping or out of the house), as for some reason automatic throttling isn't working in Arch. I started a render at the lowest performance level of 18%, (800MHz), then cranked it to 50% (2GHz), then cranked it to 100% (4GHz).

Melt continued to consume the same percentage of CPU time on average through these changes, and the rendering in Kdenlive sped up as a result. Weird!

Not weird. That sounds normal.

Allow me to explain a little differently. If I start a render with the CPU performance set to 100% and leave it there, it takes, say, 5 minutes to render a particular video. If I start the render job with the CPU performance at it's lowest setting, and then increase it, the render of the same video completes in less time, perhaps 3-4 minutes total.
User avatar Steve Guilford
Registered Member
Posts
151
Karma
0

Re: Kdenlive doesn't max the CPU?

Wed Jul 30, 2014 3:17 pm
Ranko Kohime wrote:Allow me to explain a little differently. If I start a render with the CPU performance set to 100% and leave it there, it takes, say, 5 minutes to render a particular video. If I start the render job with the CPU performance at it's lowest setting, and then increase it, the render of the same video completes in less time, perhaps 3-4 minutes total.


That sounds weirder !!! I'm at a loss to explain that. Too many factors to consider.
markoc
Registered Member
Posts
323
Karma
0

Re: Kdenlive doesn't max the CPU?

Mon Aug 04, 2014 2:12 pm
Answering to the first posting above:

Video processing is also a very memory intensive application. When you have a pipeline of several processing steps, the data between the steps in most cases goes through memory buffers.
Consider that a HD video frame is bigger than most caches!

Eight cores on a single bus / memory are already pushing the bus/memory bandwidth a lot (even the common part of the cache)!
The 8 cores might work full throttle with some compute intensive but less memory intensive scientific application, where the data fits into the "private" cache of each core, but video processing might easily hit the bus/memory bandwidth limits.

--------------------------------------------------------------------

One thing I forgot is, that the bus wait cycles probably don't show up as idle time on the OS level.
User avatar Ranko Kohime
Registered Member
Posts
41
Karma
0
OS

Re: Kdenlive doesn't max the CPU?

Sun Aug 10, 2014 6:41 pm
markoc wrote:Answering to the first posting above:

Video processing is also a very memory intensive application. When you have a pipeline of several processing steps, the data between the steps in most cases goes through memory buffers.
Consider that a HD video frame is bigger than most caches!

Eight cores on a single bus / memory are already pushing the bus/memory bandwidth a lot (even the common part of the cache)!
The 8 cores might work full throttle with some compute intensive but less memory intensive scientific application, where the data fits into the "private" cache of each core, but video processing might easily hit the bus/memory bandwidth limits.

--------------------------------------------------------------------

One thing I forgot is, that the bus wait cycles probably don't show up as idle time on the OS level.

I would think that would be reported as I/O wait? Or does I/O wait only apply for storage?
User avatar TheDiveO
Registered Member
Posts
203
Karma
2

Re: Kdenlive doesn't max the CPU?

Sun Aug 10, 2014 7:46 pm
I/O is only for disk, usb, etc input/output. Everything that officially gets input or output, so the operating system does notice. To the best of my knowledge, bus wait cycles are not considered I/O in this particular sense. Otherwise you would need to add in also every memory access too. And then graphics ... you don't want that getting included to. :)


TheDiveO on YouTube: diving videos and aerial (quadcopter) videos, all made with Kdenlive.
TheDiveO Blog: know-how about Protune-grading in Kdenlive, GoPro HD Hero 3, DJI Phantom Quadcopter, full face mask diving, and some more...
ddennedy
Registered Member
Posts
1314
Karma
0

Re: Kdenlive doesn't max the CPU?

Tue Aug 12, 2014 5:25 pm
Hi guys, interesting discussion. MLT has an option to parallelize the processing of frames across multiple threads. This is controlled by setting the real_time property on the consumer[1].
In a custom Kdenlive render profile you can add something like "real_time=-4" if you want to experiment with this. Some previous versions (? - 0.9.6) of Kdenlive exposed this for both rendering and the timeline monitor in Kdenlive Settings > Environment > Processing Threads (or something like that). However, it was removed with the claim that "it never really worked." It does work, however, with some limitations. The implementation is not optimized, and there are some effects' routines that are not thread-safe and have to be marked as such [2] or modified [3], which is not always easy. When an effect's core, heavy routine is marked not thread-safe [4], then work cannot be parallelized at this point in the processing pipeline. This creates bottlenecks. :-\ Another drawback is that if an effect is not thread-safe, but we have not yet discovered that limitation, you will likely see artifacts in the video [5].

[1] http://www.mltframework.org/bin/view/ML ... e_of_multi
[2] https://github.com/mltframework/mlt/blo ... d_safe.txt
[3] https://github.com/mltframework/mlt/com ... 342986f0ff
[4] https://github.com/mltframework/mlt/com ... c997bd7e64
[5] https://github.com/mltframework/mlt/com ... 0e25161d1c



 
Reply to topic

Bookmarks



Who is online

Registered users: Baidu [Spider], Bing [Bot], donniezazen, Exabot [Bot], Google [Bot], google01103, koriun, Majestic-12 [Bot], paviluf, ramskulls, rv8ter, Yahoo [Bot]