Extending cpuset to include user processes

[Snoopy8]

Would like to see the ability for a user to add a process to be included in cpuset. E.g. ​​would like dedicate a processor for CamillaDSP
p/s tried to manualy add cpuset but it appears that cpusets are reset at boot

[agent_kith]

Good idea. Really worth exploring CPUSet to see what it can do..

Havn't had much time to work on the new GUI past week, and unlikely to have the time with the coming long weekend. But will try and incorp this into the new GUI rework. Issue created and tracked here.

[Snoopy8]

Some more information for next release of cpuset:

1. cpuset does not stop IRQs from using the user cpuset (CPUs 2,3).  Ideally, to reduce latency, IRQs should be on a different CPU to the music player.  My workaround is to edit
   /etc/default/irqbalance and set IRQBALANCE_BANNED_CPUS=C  (hex mask value, run in CPUs 0,1 not in CPUs 2,3)
   
2. This may help with setting a new cpuset for a user process.  CamillaDSP was started as a system process at boot. Manually shifted using
   sudo cset proc --move --toset=user --pid 723 --threads
   
   Confirmed move worked using cset shield --shield -v  
   Code:

   cset: "user" cpuset of CPUSPEC(2-3) with 11 tasks running
   USER       PID  PPID SPPr TASK NAME
   -------- ----- ----- ---- ---------
   root       723     1 Sf70 [camilladsp]
   root       738     1 Sf70 [camilladsp]
   root       746     1 Sf70 [camilladsp]
   root       747     1 Sf70 [AlsaPlayback]
   root       748     1 Sf70 [AlsaCapture]
   root      2125     1 Soth [mpd]
   root      2174     1 Soth [io]
   root      2175     1 Sf40 [rtio]
   root      2178     1 Soth [player]
   root      2182     1 Soth [decoder]
   root      2387     1 Sf40 [output]
cset: done
3. Some issues with existing cpuset, a few player processes were not shifted to the user cpuset
   LMS + Squeezlite R2 : squeeze2upnp-x8  (using DLNA/UPnP bridge) remained in the system cpuset
   LMS + upmpdcli : squeeze2upnp-x8, mpd  remained in the system cpuset
   Minimserver : uses Java which remained in the system cpuset
   

Can confirm SQ improved with shifting IRQs to system cpuset and CamillaDSP to user cpuset.

*****
Added 22-Jun

A few more notes

1.  Can move a few more kernel processes from user to system cpuset using
   sudo cset shield -k on
   While I heard no discernible difference with SQ, for completeness, should include this.
   
2. With upmpdcli,  it is not possible to shift the underlying mpd process from system to user cpuset.  Given that upmpdcli is only a wrapper and the work is done by mpd, cpuset may not provide a SQ benefit for upmpdcli?
   
3. Tried dedicating a cpu to LMS and another cpu to Squeezelite R2.  No difference in SQ to running both LMS and Squeezelite R2 in same cpu.
   
4. Similarly no difference in SQ with mpd and CamillaDSP running on same cpu or on separate individual cpus.
   
5. However, with cpu intensive servers like Roon or HQplayer, there may be a benefit in running their players on a seperate cpu?  Someone should test this.
   
6. If mympd is enabled (coming in next release), should shift it from system to user cpuset. I do not use mympd and will disable it.
   

[agent_kith]

Have updated the ticket with last post. Hoping to incorp this into the new GUI in the upcoming release.

[Snoopy8]

Further adventures with cpuset; information may be useful for the further extension of cpuset within SnakeOil.

Have a 4 CPU Intel NUC and used this script for my mpd setup
 

Code:

#
# set up cpusets
#
cset set -c 0 -s system
cset set -c 1 -s irq --cpu_exclusive
cset set -c 2 -s player --cpu_exclusive
cset set -c 3 -s server --cpu_exclusive
#
# move music server and player threads
#
cset proc --move --toset=player --pid $(pgrep mpd) --threads
cset proc --move --toset=server --pid $(pgrep camilladsp) --threads
#
# move all unbound kernel threads to system cpuset, and usb irq to irq cpuset
#
cset proc -k -f root -t system
cset proc --move --toset=irq --pid $(pgrep irq/123-xhci_hc) --kthread --force

Mpd and CamiilaDSP were moved to individual CPUs using a command similar to this:
"cset proc --move --toset=player --pid $(pgrep mpd) --threads".

Ideally, to reduce latency, IRQs should be on a different CPU to the music threads. Some IRQs were bound to CPUs, but many of the IRQs were moved using the following:

1. Edited /etc/default/irqbalance and set IRQBALANCE_BANNED_CPUS=C  (hex mask value, run in CPUs 0,1 not in CPUs 2,3). 
   
2. Alternately, disabled or uninstalled IRQBalance and more IRQs will run on the first CPU (CPU0)
   
3. Used "cset proc -k -f root -t system" to move some more IRQs
   
4. Used "cset proc --move --toset=irq --pid $(pgrep irq/123-xhci_hc) --kthread --force" to move USB IRQ to its own CPU.
   

Some music threads could not be moved eg. Minimserver is Java based. These music threads should be started within a cpuset using a command similar to this:
"cset proc -s server -e /bin/bash -- /var/www/players/minimserver/start.sh"

Some useful information on cpusets
https://documentation.suse.com/sle-rt/15...model.html
https://documentation.suse.com/sle-rt/15...puset.html

[Snoopy8]

Part III.   Had the following CPUsets (Intel NUC has 4 CPUs)
 

Code:

$cset set -l
cset:
         Name       CPUs-X    MEMs-X Tasks Subs Path
------------ ---------- - ------- - ----- ---- ----------
         root        0-3 y       0 y   207    4 /
       player          2 y       0 n     7    0 /player
       system          0 n       0 n    31    0 /system
       server          3 y       0 n     5    0 /server
          irq          1 y       0 n     1    0 /irq

The player CPUset ran the following mpd processes:
 

Code:

$ cset proc -l -s player
cset: "player" cpuset of CPUSPEC(2) with 7 tasks running
USER       PID  PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root       401     2 Sr55 [irq/127-eth0]
root      1082     1 Sr75 [mpd]
root      1083     1 Sr75 [io]
root      1084     1 Sr75 [rtio]
root      1090     1 Sr75 [player]
root      1091     1 Sr75 [decoder]
root      1379     1 Sr75 [output]

The server CPUset was running DSP:
 

Code:

$ cset proc -l -s server
cset: "server" cpuset of CPUSPEC(3) with 5 tasks running
USER       PID  PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root       589     1 Sr80 [camilladsp]
root       628     1 Sr80 [camilladsp]
root       631     1 Sr80 [camilladsp]
root       632     1 Sr80 [AlsaPlayback]
root       633     1 Sr80 [AlsaCapture]

The IRQ CPUset was running the USB output:
 

Code:

$ cset proc -l -s irq
cset: "irq" cpuset of CPUSPEC(1) with 1 task running
USER       PID  PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
root        87     2 Sr85 [irq/124-xhci_hc]

Note that processes had increasing proprities down the music chain. IRQ for Ethernet: 55, feeding Mpd: 75, then processed by CamillaDSP:80 and output to IRQ for USB: 85. This ensured that downstream processes were always ready to accept inputs.

With IRQBalance removed, IRQs ran in CPU0 (System CPUset). The USB IRQ, which was the most CPU intensive task, ran on its own CPU. The Ethernet IRQ, being the input for mpd, ran on the same CPU. Further optimisation was done by moving some kernel threads to CPU0 using "cset proc -k -f root -t system".

In addition, had a 5.10 RT Kernel (built my own), did BIOS tweaks (removed hyperthreading, virtual machine settings, unnecessary ports & features) and ran DSP (convolution filters).

With mpd playing music, ran "cyclictest -t4 -p 90 -N -s -i 10000 -l 10000 -q" twenty times, with the following results.

   

Maximum latencies were typically 5 to 7 µs, which is excellent. CPU 0 had three 8 µs and one 10 µs max latency, which is not an issue because it ran everything not directly associated with music playback. However, CPU 1 (USB IRQ) and CPU 2 (mpd) had one 9 µs max latency, which suggests there could be room for improvement?  Will discuss things that require further investigation in a follow up post.

In summary, delighted with the tweaks that delivered SQ improvement for SnakeOil.

[Snoopy8]

A note for AK when CPUset capabilities are further enhanced.  Using Ubuntu 20.04, SnakeOil 1.2.9

With Upmpdcli, used the following to shift it
 

Code:

cset proc --move --toset=player --pid $(pgrep upmpdcli) --threads

The upmpdcli processes are successfully shifted to the player CPUset
 

Code:

$ cset proc -l -s player
cset: "player" cpuset of CPUSPEC(2) with 16 tasks running
USER       PID  PPID SPPr TASK NAME
-------- ----- ----- ---- ---------
upmpdcli  1139     1 Sr60 [upmpdcli]
upmpdcli  1141     1 Sr60 [upmpdcli]
upmpdcli  1145     1 Sr60 [upmpdcli]
upmpdcli  1149     1 Sr60 [upmpdcli]
upmpdcli  1150     1 Sr60 [MHD-listen]
upmpdcli  1151     1 Sr60 [MHD-listen]
upmpdcli  1153     1 Sr60 [upmpdcli]
upmpdcli  1597     1 Sr60 [upmpdcli]
upmpdcli  1599     1 Sr60 [upmpdcli]
upmpdcli  1689     1 Sr60 [upmpdcli]
upmpdcli  5595     1 Sr60 [upmpdcli]
upmpdcli  5638     1 Sr60 [upmpdcli]
upmpdcli  5797     1 Sr60 [upmpdcli]
upmpdcli  5798     1 Sr60 [upmpdcli]
upmpdcli  5840     1 Sr60 [upmpdcli]
upmpdcli  5843     1 Sr60 [MHD-connection]

Mpd have the following PIDs and should be in player CPUset (see above)
 

Code:

$ pgrep mpd
1139
1163

However, Top shows that Mpd can run in all the CPUs
 

Code:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND         P
     87 root     -86   0       0      0      0 S  10.0   0.0   2:18.91 irq/122-xhci_hc 1
   1163 root     -61   0  424452  23240  11960 S   6.0   0.1   1:29.56 mpd             3
   1139 upmpdcli -61   0 1749136  10220   7796 S   0.7   0.0   0:02.04 upmpdcli        2

Processor 2 is for Player CPUset and upmpdcli is correctly running in it.  But Mpd is appearing in processor 3 in this snapshot (and different processors in other snapshots).

Note that Mpd running on its own can be assigned to a CPUset.   It appears that the embedded Mpd process in upmpdcli cannot be locked into a CPUset. No idea why...