This is more of a question as to the why rather than a "how do I fix this?".

I recently built a new computer, with an Intel i7-4790k. So, ehm, yeah, 1080p videos now render about 200% of their playback speed.

But curiously, Kdenlive doesn't max out any of the cores, or the hyperthreads, according to any of the monitors I have available (GKrellM, KSysGuard, Htop), even when set to 8 threads in the render window. (And to be clear, yes the render window, not the settings window)

I ruled out disk throttling by placing the source and destination videos in a RAM fs, but it didn't seem to have any effect, and thermal throttling isn't the issue either, as the CPU could not exceed 60c during rendering, while during a Prime95 torture test it can get MUCH hotter.

Would anyone be able to shed light on this odd behavior so my curiosity can stop bugging me?

For reference, I've also used an i3-2310m, and an AMD A8-6600k for rendering, (and the later a quad-core overclocked to 4.5GHz), and both of these Kdenlive was able to max-out.

Same over here, on an i7 3.something GHz. I've once asked Dan Dennedy of MLT fame about this as this is about MLT multithreading and not Kdenlive. His answer was slightly disappointing (to me) and basically to the point that MLT can't max out thread usage anayway so there will always be some considerable amount of idleness. At least on more powerful systems with many cores. He doesn't plan on changing this situation, as all efforts are currently invested on movit instead and this is much more promising, especially for users with less powerful CPU hardware.

So, yes, we users with higher end i7 systems are somehow left in the cold as MLT cannot drive octocores at full load. I've noticed similar behavior to what you describe: my much lesser i5 quadcore gets maxed out, while the shiny octocore i7 won't. Damn, now we're not the coolest guys in town ... I'm afraid.

I think I know why. Audio effects are tight to real time. So, it must play at normal speed. One thing you, can do - run two or more renderers

kgolubev wrote:I think I know why. Audio effects are tight to real time. So, it must play at normal speed. One thing you, can do - run two or more renderers

I would actually tend to doubt that. My i3 laptop had no trouble rendering, even though some of my 1/2-hour long videos were taking upwards of 6 hours to render, and on the other hand, even though my CPU isn't being maxed out, I'm still rendering in half the runtime of the video on the new rig.

Bear in mind Linux is a time-shared O/S - not a RTOS (e.g. QNX or RT-11...that's ancient !!!). Therefore, the OS tries to prevent applications from maxing out the CPU by scheduling jobs at regular intervals - whether they like it or not. A process can 'nice' it's way to a higher priority but it's still up to whims of the OS's scheduler as to when you get to play with the CPU. If you want performance then you'd use runlevel 3, eliminate unneeded processes and a deploy a headless rendering engine.

Nope, as Dan Dennedy explained it is difficult to even distribute work among eight threads in MLT. This is not a RTOS or any other OS scheduling problem. This is an application work distribution problem inside the application. Please don't mix up completely separate things here, unless you are clearly joking. But then, the irony must have slipped past me...

So, weird experiment produces weird results: I've been manually throttling my CPU when I plan to leave it alone for a while (while sleeping or out of the house), as for some reason automatic throttling isn't working in Arch. I started a render at the lowest performance level of 18%, (800MHz), then cranked it to 50% (2GHz), then cranked it to 100% (4GHz).

Melt continued to consume the same percentage of CPU time on average through these changes, and the rendering in Kdenlive sped up as a result. Weird!

Ranko Kohime wrote:So, weird experiment produces weird results: I've been manually throttling my CPU when I plan to leave it alone for a while (while sleeping or out of the house), as for some reason automatic throttling isn't working in Arch. I started a render at the lowest performance level of 18%, (800MHz), then cranked it to 50% (2GHz), then cranked it to 100% (4GHz).

Melt continued to consume the same percentage of CPU time on average through these changes, and the rendering in Kdenlive sped up as a result. Weird!

Not weird. That sounds normal.

Steve Guilford wrote:

Ranko Kohime wrote:So, weird experiment produces weird results: I've been manually throttling my CPU when I plan to leave it alone for a while (while sleeping or out of the house), as for some reason automatic throttling isn't working in Arch. I started a render at the lowest performance level of 18%, (800MHz), then cranked it to 50% (2GHz), then cranked it to 100% (4GHz).

Melt continued to consume the same percentage of CPU time on average through these changes, and the rendering in Kdenlive sped up as a result. Weird!

Not weird. That sounds normal.

Allow me to explain a little differently. If I start a render with the CPU performance set to 100% and leave it there, it takes, say, 5 minutes to render a particular video. If I start the render job with the CPU performance at it's lowest setting, and then increase it, the render of the same video completes in less time, perhaps 3-4 minutes total.

Ranko Kohime wrote:Allow me to explain a little differently. If I start a render with the CPU performance set to 100% and leave it there, it takes, say, 5 minutes to render a particular video. If I start the render job with the CPU performance at it's lowest setting, and then increase it, the render of the same video completes in less time, perhaps 3-4 minutes total.

That sounds weirder !!! I'm at a loss to explain that. Too many factors to consider.

Answering to the first posting above:

Video processing is also a very memory intensive application. When you have a pipeline of several processing steps, the data between the steps in most cases goes through memory buffers.
Consider that a HD video frame is bigger than most caches!

Eight cores on a single bus / memory are already pushing the bus/memory bandwidth a lot (even the common part of the cache)!
The 8 cores might work full throttle with some compute intensive but less memory intensive scientific application, where the data fits into the "private" cache of each core, but video processing might easily hit the bus/memory bandwidth limits.

--------------------------------------------------------------------

One thing I forgot is, that the bus wait cycles probably don't show up as idle time on the OS level.

markoc wrote:Answering to the first posting above:

Video processing is also a very memory intensive application. When you have a pipeline of several processing steps, the data between the steps in most cases goes through memory buffers.
Consider that a HD video frame is bigger than most caches!

Eight cores on a single bus / memory are already pushing the bus/memory bandwidth a lot (even the common part of the cache)!
The 8 cores might work full throttle with some compute intensive but less memory intensive scientific application, where the data fits into the "private" cache of each core, but video processing might easily hit the bus/memory bandwidth limits.

--------------------------------------------------------------------

One thing I forgot is, that the bus wait cycles probably don't show up as idle time on the OS level.

I would think that would be reported as I/O wait? Or does I/O wait only apply for storage?

I/O is only for disk, usb, etc input/output. Everything that officially gets input or output, so the operating system does notice. To the best of my knowledge, bus wait cycles are not considered I/O in this particular sense. Otherwise you would need to add in also every memory access too. And then graphics ... you don't want that getting included to.

Hi guys, interesting discussion. MLT has an option to parallelize the processing of frames across multiple threads. This is controlled by setting the real_time property on the consumer[1].
In a custom Kdenlive render profile you can add something like "real_time=-4" if you want to experiment with this. Some previous versions (? - 0.9.6) of Kdenlive exposed this for both rendering and the timeline monitor in Kdenlive Settings > Environment > Processing Threads (or something like that). However, it was removed with the claim that "it never really worked." It does work, however, with some limitations. The implementation is not optimized, and there are some effects' routines that are not thread-safe and have to be marked as such [2] or modified [3], which is not always easy. When an effect's core, heavy routine is marked not thread-safe [4], then work cannot be parallelized at this point in the processing pipeline. This creates bottlenecks. Another drawback is that if an effect is not thread-safe, but we have not yet discovered that limitation, you will likely see artifacts in the video [5].

[1] http://www.mltframework.org/bin/view/ML ... e_of_multi
[2] https://github.com/mltframework/mlt/blo ... d_safe.txt
[3] https://github.com/mltframework/mlt/com ... 342986f0ff
[4] https://github.com/mltframework/mlt/com ... c997bd7e64
[5] https://github.com/mltframework/mlt/com ... 0e25161d1c

Time to update this thread (in the idle hope that forum visitors actually use the search function instead of boring everyone to death asking the same question he guy two posts before already asked and got answered).

The parameter Dan Dennedy mentioned is not (yet?) exposed in Kdenlive. The number of "Encoder threads" in the rendering dialog controls only the "threads" parameter of MLT.

Do not mix the "Encoder threads" settings in the rendering dialog with the "Processing threads" in Kdenlive's configure dialog, section "Environment". The latter controls the threads used in Kdenlive during editing, but not rendering. However, on fast multicore machines using more than a single Processing thread causes a the timeline cursor to not correctly update. This is probably not a problem in MLT, but instead in Kdenlive.