[split] Real Time Kernels discussion

[hkphantomgtr]

For the change in sound from low latency kernel to RT kernel, even the difference is not as significant as change from ordinary kernel to low latency kernel, it's still clear and enjoyable. 

In comparing with the sound of low latency kernel, RT kernel further refines the focus, positioning, and also the dimension of the sound stage.  The sound stage is just bigger, wider and deeper.  It simply make the sound more holographic.

RT kernel also further enhances the details, texture, and dynamics.  That makes the sound more vivid.  It's more impacting and influencing, closer to the live sound.

Is there any more feedback about the change in sound after using latest RT kernel in your OS system?

[Snoopy8]

@uglymusic let's continue the discussion on Real Time (RT) kernel for Pi here.  Some history first.

Prior to SO 1.2.x, AK delivered RT SO images but this was not sustainable. SO 1.2.x became Ubuntu based with standard kernel until the arrival of the Ubuntu RT kernel this year.  However the RT kernel was for X86, not Pi.

As an aside, AK had an ambitious project to customise RT kernels for individuals but it stalled.
https://www.snakeoil-os.net/forums/Threa...el+builder
If the project is revived, I would like to see RT kernels sold for profit because it is resource intensive and SO could do with income.

For the Pi, start with this
https://forums.raspberrypi.com/viewtopic.php?t=344994

Then, use the info from here to look at the Make parameters
https://docs.ros.org/en/foxy/Tutorials/M...ROS-2.html
Try removing the unnecessary parameters for wireless, GPU, crypto, foreign hardware etc. This edit is slow but it can reduce the Make  process time by a lot.  However, you can also cause the boot to hang if too much is removed.

I have not tried with the Pi, but I did find problems with the ROS guide and other web resources. I will try to help but am illiterate in Linux. I can Google my way to to find potential fixes, but cannot guarantee that things will work.

*** Forgot to add this important point.  I backup my SO instance using Clonezilla before starting.  Things will and do go wrong. Recommend cloning SD card before starting.

*** A difference between the Intel NUC and Pi is the tweaking of the BIOS.  In the NUC, can switch off unnecessary things like Hyperthreading, virtualisation, power savings, all of which can impact SQ.  There are no BIOS tweaks for the Pi.  May have to tweak things in Make?

[uglymusic]

@uglymusic let's continue the discussion on Real Time (RT) kernel for Pi here.  Some history first.

Prior to SO 1.2.x, AK delivered RT SO images but this was not sustainable. SO 1.2.x became Ubuntu based with standard kernel until the arrival of the Ubuntu RT kernel this year.  However the RT kernel was for X86, not Pi.

As an aside, AK had an ambitious project to customise RT kernels for individuals but it stalled.
https://www.snakeoil-os.net/forums/Threa...el+builder
If the project is revived, I would like to see RT kernels sold for profit because it is resource intensive and SO could do with income.

For the Pi, start with this
https://forums.raspberrypi.com/viewtopic.php?t=344994

Then, use the info from here to look at the Make parameters
https://docs.ros.org/en/foxy/Tutorials/M...ROS-2.html
Try removing the unnecessary parameters for wireless, GPU, crypto, foreign hardware etc. This edit is slow but it can reduce the Make  process time by a lot.  However, you can also cause the boot to hang if too much is removed.

I have not tried with the Pi, but I did find problems with the ROS guide and other web resources. I will try to help but am illiterate in Linux. I can Google my way to to find potential fixes, but cannot guarantee that things will work.

*** Forgot to add this important point.  I backup my SO instance using Clonezilla before starting.  Things will and do go wrong. Recommend cloning SD card before starting.

*** A difference between the Intel NUC and Pi is the tweaking of the BIOS.  In the NUC, can switch off unnecessary things like Hyperthreading, virtualisation, power savings, all of which can impact SQ.  There are no BIOS tweaks for the Pi.  May have to tweak things in Make?

Thank you, @Snoopy8. This looks like the pointer I need, although I suspect it'll be Christmas reading and possibly another opportunity for indigestion 

[Snoopy8]

Thank you, @Snoopy8. This looks like the pointer I need, although I suspect it'll be Christmas reading and possibly another opportunity for indigestion 

Building an RT kernel is not for the faint hearted and very time consuming.

I have built a new RT kernel for my NUC, using the latest stable 6.1 RT version.   It is compatible with Ubuntu 22.04 and sounds better than the standard Ubuntu RT kernel 5.15.0-1032-realtime.  Maybe it is because the 6.1 kernel sounds better, but more likely due to me taking away unnecessary hardware options.  There are a lot more config options in Ubuntu now than a few years ago and took an hour to compile.   If I was not in a hurry, I could have removed a lot more config options to reduce the compile time.

I have also tried doing for the Pi.  The good news is the steps in the link I mentioned works
https://forums.raspberrypi.com/viewtopic.php?t=344994
Just have to change to the correct versions, suggest using 6.1.67. 

The bad news is that it took 1.5 hours to compile on a CM4, and likely many, many hours for your Pi3.  It is possible to use an external box to cross compile, but I have not gone there yet.  And the instructions will generate a standalone kernel, which used to be the default way for the Pi.  However, in bookworm for the CM4, it is using initramfs which should allow multiple kernels, but not working for me.  I have posted this info in the 1.3 beta thread and hope AK can give me answer.

If all of this is too much, suggest continuing your Christmas celebrations. Not only will this cause indigestion, but also headaches, frustration, stress 

[uglymusic]

Thank you, @Snoopy8. This looks like the pointer I need, although I suspect it'll be Christmas reading and possibly another opportunity for indigestion 

Building an RT kernel is not for the faint hearted and very time consuming.

I have built a new RT kernel for my NUC, using the latest stable 6.1 RT version.   It is compatible with Ubuntu 22.04 and sounds better than the standard Ubuntu RT kernel 5.15.0-1032-realtime.  Maybe it is because the 6.1 kernel sounds better, but more likely due to me taking away unnecessary hardware options.  There are a lot more config options in Ubuntu now than a few years ago and took an hour to compile.   If I was not in a hurry, I could have removed a lot more config options to reduce the compile time.

I have also tried doing for the Pi.  The good news is the steps in the link I mentioned works
https://forums.raspberrypi.com/viewtopic.php?t=344994
Just have to change to the correct versions, suggest using 6.1.67. 

The bad news is that it took 1.5 hours to compile on a CM4, and likely many, many hours for your Pi3.  It is possible to use an external box to cross compile, but I have not gone there yet.  And the instructions will generate a standalone kernel, which used to be the default way for the Pi.  However, in bookworm for the CM4, it is using initramfs which should allow multiple kernels, but not working for me.  I have posted this info in the 1.3 beta thread and hope AK can give me answer.

If all of this is too much, suggest continuing your Christmas celebrations. Not only will this cause indigestion, but also headaches, frustration, stress 

But, on my RPi 5, compiling will be quicker! (When I get back to running SO on it, of course).

I'll only know the full horror once I start to experience it! 

Happy holidays to me 

[agent_kith]

Then, use the info from here to look at the Make parameters
https://docs.ros.org/en/foxy/Tutorials/M...ROS-2.html

Segway a bit. But that's my day job now. ROS . Day job is taking away all my time for now as it's kind of make or break situation at the moment. Interesting to see this mentioned here.

Technically I'm not dealing with ROS, but have to past couple of months as I have to spend time to clean up poor quality code left by a parasite.

[Snoopy8]

I have also tried doing for the Pi.  The good news is the steps in the link I mentioned works
https://forums.raspberrypi.com/viewtopic.php?t=344994
Just have to change to the correct versions, suggest using 6.1.67. 

A note for those who may want to try building an RT kernel for Bookworm. I missed this critical point:

Prior to Bookworm, Raspberry Pi OS stored the boot partition at /boot/. Since Bookworm, the boot partition is located at /boot/firmware/.

https://www.raspberrypi.com/documentatio...ile-format

Hence, it is likely that information on the web for the Pi boot process is pre Bookworm.  For example, config.txt is used to configure the boot, but you should be changing /boot/firmware/config.txt, NOT /boot/config.txt (which is still there!).

[Snoopy8]

After a week of trial and error (and lots of errors including a failed boot!), I have a working version of a 6.1 RT kernel for the Pi.  It is based on replacing the Pi 4 kernel8.img with the RT version.  Still no success with getting the initramfs working, but I suspect it is a combination of editing /boot/firmware/config.txt and maybe change Grub???

No doubt that the RT kernel sounds better.  And the cyclictest numbers prove it, with cpuset enabled (user running on last 2 CPUs), and mpd playing music.
 

Code:

Generic Kernel
sudo cyclictest -t4 -p 90 -N -s -i 10000 -l 10000 -q
# /dev/cpu_dma_latency set to 0us
T: 0 ( 2247) P:90 I:10000 C:  10000 Min:  11528 Act:   28286 Avg:   30429 Max:  878427
T: 1 ( 2248) P:90 I:10500 C:   9533 Min:  11171 Act:   24541 Avg:   24396 Max:   67930
T: 2 ( 2249) P:90 I:11000 C:   9094 Min:  11630 Act:   30500 Avg:   31576 Max:  493441
T: 3 ( 2250) P:90 I:11500 C:   8706 Min:  10571 Act:   26922 Avg:   24326 Max:   63533

RT Kernel
sudo cyclictest -t4 -p 90 -N -s -i 10000 -l 10000 -q
# /dev/cpu_dma_latency set to 0us
T: 0 ( 1797) P:90 I:10000 C:  10000 Min:   9734 Act:   25065 Avg:   24552 Max:   63429
T: 1 ( 1798) P:90 I:10500 C:   9529 Min:   9631 Act:   24415 Avg:   22432 Max:   48408
T: 2 ( 1799) P:90 I:11000 C:   9095 Min:   9978 Act:   27283 Avg:   24627 Max:   52668
T: 3 ( 1800) P:90 I:11500 C:   8702 Min:   9763 Act:   26262 Avg:   22468 Max:   60311

For people not familiar with using the Cyclictest, the only important number is max latency.  All times are in microseconds.  The lower the max latency, the better for SQ.  With the generic kernel, it is all over the place and can be as long as 1,000 milliseconds (1 second).  With the RT kernel, the times are more consistent, with max latency around  50 to 70 milliseconds.  A good result.

[Snoopy8]

@uglymusic, this my Xmas present which may or may not cause indigestion ?     Use these series of commands to create a RT kernel called kernel8. 
 

Code:

# based on https://forums.raspberrypi.com/viewtopic.php?t=344994
uname -r           # show existing kernel
sudo apt update
sudo apt install git bc bison flex libssl-dev make libncurses5-dev  
mkdir kernel
cd kernel/
git clone --depth=1 --branch rpi-6.1.y https://github.com/raspberrypi/linux
# check this website for correct version of RT kernel
# https://wiki.linuxfoundation.org/realtime/start
wget https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.1/patch-6.1.67-rt20.patch.gz
cd linux/
zcat ../patch-6.1.67-rt20.patch.gz | patch -p1 --dry-run    #check the patch fits
zcat ../patch-6.1.67-rt20.patch.gz | patch -p1
make bcm2711_defconfig
make menuconfig                #General -> Preemption Model select Fully Preemptible Kernel (Real-Time)
nano .config                        #add personalised suffix to CONFIG_LOCALVERSION, optional
scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS

# ideally run next command on terminal because it takes time; SSH session may reset. Remember to cd kernel/linux
make -j4 Image.gz modules dtbs  # -j4 uses 4 CPUs, prompted for certificate, enter, cpuset should be turned off
sudo make -j4 modules_install
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/
sudo cp /boot/firmware/kernel8.img /boot/firmware/kernel8old.img
sudo cp arch/arm64/boot/Image.gz /boot/firmware/kernel8.img

sudo reboot
uname -r                # confirm new kernel

It will take just over 2 hours to do the compile (make -j4...).

@hkphantomgtr, you have been testing various kernels for the Pi.  Not sure whether you want to try building a RT kernel???

[hkphantomgtr]

@uglymusic, this my Xmas present which may or may not cause indigestion ?     Use these series of commands to create a RT kernel called kernel8. 
 

Code:

# based on https://forums.raspberrypi.com/viewtopic.php?t=344994
uname -r           # show existing kernel
sudo apt update
sudo apt install git bc bison flex libssl-dev make libncurses5-dev  
mkdir kernel
cd kernel/
git clone --depth=1 --branch rpi-6.1.y https://github.com/raspberrypi/linux
# check this website for correct version of RT kernel
# https://wiki.linuxfoundation.org/realtime/start
wget https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.1/patch-6.1.67-rt20.patch.gz
cd linux/
zcat ../patch-6.1.67-rt20.patch.gz | patch -p1 --dry-run    #check the patch fits
zcat ../patch-6.1.67-rt20.patch.gz | patch -p1
make bcm2711_defconfig
make menuconfig                #General -> Preemption Model select Fully Preemptible Kernel (Real-Time)
nano .config                        #add personalised suffix to CONFIG_LOCALVERSION
scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS

# ideally run next command on terminal because it takes time; SSH session may reset
make -j4 Image.gz modules dtbs  # -j4 uses 4 CPUs, prompted for certificate, enter, cpuset should be turned off
sudo make -j4 modules_install
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/
sudo cp /boot/firmware/kernel8.img /boot/firmware/kernel8old.img
sudo cp arch/arm64/boot/Image.gz /boot/firmware/kernel8.img

sudo reboot
uname -r                # confirm new kernel

It will take just over 2 hours to do the compile (make -j4...).

@hkphantomgtr, you have been testing various kernels for the Pi.  Not sure whether you want to try building a RT kernel???

I have played around building RT kernel for CAS 10 years ago.  By that time there was a software, I just ticked, unticked, ticked, unticked thru a long menu, then press confirm for several times, after the compilation then the RT kernel is done.  But I didn't do it for such long time that I've forgot everything about it.

I'm afraid that, unless I found a similar software, I may not be able to follow those steps in the web.

I'd rather leave this to you and AK.