[Rtai] isolcpus

[Jan Benda]

Hi,

recently I played around with the isolcpus kernel parameter (with great
success - see below). In this context I have two questions:

1. Right now I select a CPU with the isolcpus kernel parameter and then
I also need to select the same CPU for the real time task / latency test.

Is there a way to figure out which CPUs are isolated and then use this
information for making the RTAI task run on that CPU (via
rt_set_runnable_on_cpus as in kern/latency test)?

2. I need to edit the latency-module.c file to change the CPU id on
which the test is run (modifying the RUNNABLE_ON_CPUS define). It would
be very nice if this would be a parameter that can be set when loading
the module - or as of 1) if the tests would be run on the isolated CPU.
Same for all the other tests in testsuite.

Here are the kern/latency tests with isolcpus=1 and latency test running
on CPU1:

Idle condition: average jitter (lat max - lat min) 250ns, maximum < 1000ns

Under full cpu, memory, io, and network load: average jitter 1200ns, max
about 2000ns

On a Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz with RTAI 5.1,
hal-linux-4.4.115-x86-10.patch. idle-poll and tsc-reliable.

Jan

[Jan Benda]

Hi Paolo,

First, let me say that it is likely you have a very good system. In fact
your performances without cpu isolation  I somethin I would be dreaming
for. Even without creating any particular background Linux load, on the
best system I have, I get 3-4 times your values.
That said, CPU isolation, appropriately including the other two option
you mentions, is doing its duty.

Yes, I am actually thrilled by those latencies. CPU isolation on this
machine cuts the latencies by almost a factor two (both on an idle one
and under load).

1 - The kernel isolcpus are those that you should use for RTAI. So there
is nothing to else to guess and you should put your latency task there.
Take into account that rt_set_runnable_on_cpus will put your task on the
least used cpu within those you allowed. Naturally if the mask allows
more than one. If you have a single cpu of choice use
rt_set_runnable_on_cpuid instead. Moreover, you could use
rt_task_init_cpuid, in place of rt_task_init, and have your task placed
on your cpu of choice from its very beginning.

Yes, that I understand. What I want, however, that I isolate a CPU with
the kernelparameter, eg. isolcpus=2, and then have my kernel module
automatically select the isolated CPU for running the RTAI task.

When RTAI modules are loaded (without any parameter), then rtai_hal
notices the isolation mask from the linux kernel. Does this already
influence how RTAI tasks are handled? In particular, if rtai_hal knows
that CPU 2 is isolated, where will a rt_task_init job (without cpuid
specified) be executed, will it go on CPU 2? If yes then I am happy.

If no then, what I wanted to know is, how can I write some code in my
kernel module with an RTAI task, that reads the isolcpus mask, so that I
can pass it directly on to rt_set_runnable_on_cpus, or the corresponding
rt_task_init_cpu* function. Such that I do not have to change my code
when I play around with the isolcpus kernel parameter.

2) For sake of simplicity, assuming you have elected a single cpu (i.e.
rt_task_init_cpuid or rt_set_runnable_on_cpuid) you can copy the module
int mycpuid = 1; // default value
RTAI_MODULE_PARM(mycpuid, int);
MODULE_PARM_DESC(mycpuid, "The cpu number I want to run on (default: 1)")
.
.
rt_task_init_cpuid(%thread, fun, 0, 8192, 0, 1, 0, mycouid);  // in
__latency_init

I wanted to suggest that somebody does this in the cvs...

It is also useful you to inform rtai_hal.ko thatbyou are isolating cpus
insmod /usr/realtime/modules/rtai_hal.ko  IsolCpusMask=0x??;
where IsolCpusMask is the bit mask of your isolcpus.
In such a way also the Linux interrupt handlers will be removed from
your isolated cpus.

Now I am puzzled. In README.ISOLCPUS you say
"Starting from RTAI-4.1 rtai_hal.ko can inheredit the cpu isolation mask
from the kernel. So there is nomore the need to set IsolCpuMask at the
insmoding of rtai_hal.ko."

So if I isolate cpus using the isolcpus kernel parameter, there is no
reason to pass it again to rtai_hal via its IsolCpusMask, right?

Jan

Cheers, Paolo.

Post by Jan Benda
Hi,
recently I played around with the isolcpus kernel parameter (with
1. Right now I select a CPU with the isolcpus kernel parameter and
then I also need to select the same CPU for the real time task /
latency test.
Is there a way to figure out which CPUs are isolated and then use this
information for making the RTAI task run on that CPU (via
rt_set_runnable_on_cpus as in kern/latency test)?
2. I need to edit the latency-module.c file to change the CPU id on
which the test is run (modifying the RUNNABLE_ON_CPUS define). It
would be very nice if this would be a parameter that can be set when
loading the module - or as of 1) if the tests would be run on the
isolated CPU. Same for all the other tests in testsuite.
Here are the kern/latency tests with isolcpus=1 and latency test
Idle condition: average jitter (lat max - lat min) 250ns, maximum < 1000ns
Under full cpu, memory, io, and network load: average jitter 1200ns,
max about 2000ns
hal-linux-4.4.115-x86-10.patch. idle-poll and tsc-reliable.
Jan
_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

--
----------------------------------------------------------------------

Prof. Dr. Jan Benda
Abteilung Neuroethologie
Institut fuer Neurobiologie
Universitaet Tuebingen
Auf der Morgenstelle 28 E phone: +49 / 7071 - 29 74573
72076 Tuebingen fax: +49 / 7071 - 29 5144

----------------------------------------------------------------------

[Paolo Mantegazza]

You are right about the fact that there is no more the need to insmod
rtai_hal.ko stating the value of IsolCpuMask.
So I should have been clearer in my statement. The point is that it has
been left to give the possibility of checking isolcpus, as stated at
boot, with the expected one. Therefore, my suggestion was just my
practice and needs not to be followed.

As for making available the possibility of knowing the map of the cpus
isolated at boot, the matter is trivial.
Simply put the lines below in your kernel module code:

#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
#define CPU_ISOLATED_MAP (&cpu_isolated_map)
        extern unsigned long cpu_isolated_map;
#else
#define CPU_ISOLATED_MAP (cpu_isolated_map)
        extern cpumask_var_t cpu_isolated_map;
#endif

where "cpu_isolated_map" is exported from the kernel. It is simply the
boot isolcpus reflected in a bitmap.

So "cpu_isolated_map" is what you need to implement you task placing
policy using the available RTAI functions, we mentioned before.

In my view RTAI should only provide the tools to allow any user to
implement any allocation of the tasks on specific isolated cpus,
especially if they are more than one.

Paolo.

Post by Jan Benda
Hi Paolo,

First, let me say that it is likely you have a very good system. In
fact your performances without cpu isolation  I somethin I would be
dreaming for. Even without creating any particular background Linux
load, on the best system I have, I get 3-4 times your values.
That said, CPU isolation, appropriately including the other two
option you mentions, is doing its duty.

Yes, I am actually thrilled by those latencies. CPU isolation on this
machine cuts the latencies by almost a factor two (both on an idle one
and under load).

1 - The kernel isolcpus are those that you should use for RTAI. So
there is nothing to else to guess and you should put your latency
task there. Take into account that rt_set_runnable_on_cpus will put
your task on the least used cpu within those you allowed. Naturally
if the mask allows more than one. If you have a single cpu of choice
use rt_set_runnable_on_cpuid instead. Moreover, you could use
rt_task_init_cpuid, in place of rt_task_init, and have your task
placed on your cpu of choice from its very beginning.

Yes, that I understand. What I want, however, that I isolate a CPU
with the kernelparameter, eg. isolcpus=2, and then have my kernel
module automatically select the isolated CPU for running the RTAI task.
When RTAI modules are loaded (without any parameter), then rtai_hal
notices the isolation mask from the linux kernel. Does this already
influence how RTAI tasks are handled? In particular, if rtai_hal knows
that CPU 2 is isolated, where will a rt_task_init job (without cpuid
specified) be executed, will it go on CPU 2? If yes then I am happy.
If no then, what I wanted to know is, how can I write some code in my
kernel module with an RTAI task, that reads the isolcpus mask, so that
I can pass it directly on to rt_set_runnable_on_cpus, or the
corresponding
rt_task_init_cpu* function. Such that I do not have to change my code
when I play around with the isolcpus kernel parameter.

2) For sake of simplicity, assuming you have elected a single cpu
(i.e. rt_task_init_cpuid or rt_set_runnable_on_cpuid) you can copy
int mycpuid = 1; // default value
RTAI_MODULE_PARM(mycpuid, int);
MODULE_PARM_DESC(mycpuid, "The cpu number I want to run on (default: 1)")
.
.
rt_task_init_cpuid(%thread, fun, 0, 8192, 0, 1, 0, mycouid);  // in
__latency_init

I wanted to suggest that somebody does this in the cvs...

It is also useful you to inform rtai_hal.ko thatbyou are isolating
insmod /usr/realtime/modules/rtai_hal.ko  IsolCpusMask=0x??;
where IsolCpusMask is the bit mask of your isolcpus.
In such a way also the Linux interrupt handlers will be removed from
your isolated cpus.

Now I am puzzled. In README.ISOLCPUS you say
"Starting from RTAI-4.1 rtai_hal.ko can inheredit the cpu isolation
mask from the kernel. So there is nomore the need to set IsolCpuMask
at the insmoding of rtai_hal.ko."
So if I isolate cpus using the isolcpus kernel parameter, there is no
reason to pass it again to rtai_hal via its IsolCpusMask, right?
Jan

Cheers, Paolo.

Post by Jan Benda
Hi,
recently I played around with the isolcpus kernel parameter (with
1. Right now I select a CPU with the isolcpus kernel parameter and
then I also need to select the same CPU for the real time task /
latency test.
Is there a way to figure out which CPUs are isolated and then use
this information for making the RTAI task run on that CPU (via
rt_set_runnable_on_cpus as in kern/latency test)?
2. I need to edit the latency-module.c file to change the CPU id on
which the test is run (modifying the RUNNABLE_ON_CPUS define). It
would be very nice if this would be a parameter that can be set when
loading the module - or as of 1) if the tests would be run on the
isolated CPU. Same for all the other tests in testsuite.
Here are the kern/latency tests with isolcpus=1 and latency test
Idle condition: average jitter (lat max - lat min) 250ns, maximum < 1000ns
Under full cpu, memory, io, and network load: average jitter 1200ns,
max about 2000ns
hal-linux-4.4.115-x86-10.patch. idle-poll and tsc-reliable.
Jan
_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

[Jan Benda]

Hi Paolo,

thank you for your hint on the cpu_isolated_mask. With that and some
grepping that lead me to the code for rtai_hal I came up with this
solution to figure out the first isolated CPU id:

#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
extern unsigned long cpu_isolated_map;
#else
extern cpumask_var_t cpu_isolated_map;
#endif
int isolatedCPUId = -1;

...

#ifdef CONFIG_SMP
for ( i=0; i<RTAI_NR_CPUS; i++ ) {
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
if ( cpu_isolated_map & (1<<i) ) {
#else
if ( cpumask_test_cpu( i, cpu_isolated_map ) ) {
#endif
isolatedCPUId = i;
break;
}
}
#endif

...

if ( isolatedCPUId >= 0 )
rt_set_runnable_on_cpuid( &dynClampTask.rtTask, isolatedCPUId );

With that the RTAI loop runs on the first isolated CPU in case it was
specified via the kernel parameter isolcpus. Indeed, no need for RTAI to
provide any magic. Runs like a charm with the < 500ns jitter I reported
in my previous email.

Cheers,
Jan

Post by Paolo Mantegazza
You are right about the fact that there is no more the need to insmod
rtai_hal.ko stating the value of IsolCpuMask.
So I should have been clearer in my statement. The point is that it has
been left to give the possibility of checking isolcpus, as stated at
boot, with the expected one. Therefore, my suggestion was just my
practice and needs not to be followed.
As for making available the possibility of knowing the map of the cpus
isolated at boot, the matter is trivial.
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
#define CPU_ISOLATED_MAP (&cpu_isolated_map)
        extern unsigned long cpu_isolated_map;
#else
#define CPU_ISOLATED_MAP (cpu_isolated_map)
        extern cpumask_var_t cpu_isolated_map;
#endif
where "cpu_isolated_map" is exported from the kernel. It is simply the
boot isolcpus reflected in a bitmap.
So "cpu_isolated_map" is what you need to implement you task placing
policy using the available RTAI functions, we mentioned before.
In my view RTAI should only provide the tools to allow any user to
implement any allocation of the tasks on specific isolated cpus,
especially if they are more than one.
Paolo.

Post by Jan Benda
Hi Paolo,

First, let me say that it is likely you have a very good system. In
fact your performances without cpu isolation  I somethin I would be
dreaming for. Even without creating any particular background Linux
load, on the best system I have, I get 3-4 times your values.
That said, CPU isolation, appropriately including the other two
option you mentions, is doing its duty.

Yes, I am actually thrilled by those latencies. CPU isolation on this
machine cuts the latencies by almost a factor two (both on an idle one
and under load).

1 - The kernel isolcpus are those that you should use for RTAI. So
there is nothing to else to guess and you should put your latency
task there. Take into account that rt_set_runnable_on_cpus will put
your task on the least used cpu within those you allowed. Naturally
if the mask allows more than one. If you have a single cpu of choice
use rt_set_runnable_on_cpuid instead. Moreover, you could use
rt_task_init_cpuid, in place of rt_task_init, and have your task
placed on your cpu of choice from its very beginning.

Yes, that I understand. What I want, however, that I isolate a CPU
with the kernelparameter, eg. isolcpus=2, and then have my kernel
module automatically select the isolated CPU for running the RTAI task.
When RTAI modules are loaded (without any parameter), then rtai_hal
notices the isolation mask from the linux kernel. Does this already
influence how RTAI tasks are handled? In particular, if rtai_hal knows
that CPU 2 is isolated, where will a rt_task_init job (without cpuid
specified) be executed, will it go on CPU 2? If yes then I am happy.
If no then, what I wanted to know is, how can I write some code in my
kernel module with an RTAI task, that reads the isolcpus mask, so that
I can pass it directly on to rt_set_runnable_on_cpus, or the
corresponding
rt_task_init_cpu* function. Such that I do not have to change my code
when I play around with the isolcpus kernel parameter.

2) For sake of simplicity, assuming you have elected a single cpu
(i.e. rt_task_init_cpuid or rt_set_runnable_on_cpuid) you can copy
int mycpuid = 1; // default value
RTAI_MODULE_PARM(mycpuid, int);
MODULE_PARM_DESC(mycpuid, "The cpu number I want to run on (default: 1)")
.
.
rt_task_init_cpuid(%thread, fun, 0, 8192, 0, 1, 0, mycouid);  // in
__latency_init

I wanted to suggest that somebody does this in the cvs...

It is also useful you to inform rtai_hal.ko thatbyou are isolating
insmod /usr/realtime/modules/rtai_hal.ko  IsolCpusMask=0x??;
where IsolCpusMask is the bit mask of your isolcpus.
In such a way also the Linux interrupt handlers will be removed from
your isolated cpus.

Now I am puzzled. In README.ISOLCPUS you say
"Starting from RTAI-4.1 rtai_hal.ko can inheredit the cpu isolation
mask from the kernel. So there is nomore the need to set IsolCpuMask
at the insmoding of rtai_hal.ko."
So if I isolate cpus using the isolcpus kernel parameter, there is no
reason to pass it again to rtai_hal via its IsolCpusMask, right?
Jan

Cheers, Paolo.

Post by Jan Benda
Hi,
recently I played around with the isolcpus kernel parameter (with
1. Right now I select a CPU with the isolcpus kernel parameter and
then I also need to select the same CPU for the real time task /
latency test.
Is there a way to figure out which CPUs are isolated and then use
this information for making the RTAI task run on that CPU (via
rt_set_runnable_on_cpus as in kern/latency test)?
2. I need to edit the latency-module.c file to change the CPU id on
which the test is run (modifying the RUNNABLE_ON_CPUS define). It
would be very nice if this would be a parameter that can be set when
loading the module - or as of 1) if the tests would be run on the
isolated CPU. Same for all the other tests in testsuite.
Here are the kern/latency tests with isolcpus=1 and latency test
Idle condition: average jitter (lat max - lat min) 250ns, maximum < 1000ns
Under full cpu, memory, io, and network load: average jitter 1200ns,
max about 2000ns
hal-linux-4.4.115-x86-10.patch. idle-poll and tsc-reliable.
Jan
_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

[William Montgomery]

Hi Jan,

The performance numbers you report are impressive.
Would you be able to share the specific motherboard or chipset used?

Regards,
Wm

Post by Jan Benda
Hi Paolo,
thank you for your hint on the cpu_isolated_mask. With that and some
grepping that lead me to the code for rtai_hal I came up with this solution
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
extern unsigned long cpu_isolated_map;
#else
extern cpumask_var_t cpu_isolated_map;
#endif
int isolatedCPUId = -1;
...
#ifdef CONFIG_SMP
for ( i=0; i<RTAI_NR_CPUS; i++ ) {
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
if ( cpu_isolated_map & (1<<i) ) {
#else
if ( cpumask_test_cpu( i, cpu_isolated_map ) ) {
#endif
isolatedCPUId = i;
break;
}
}
#endif
...
if ( isolatedCPUId >= 0 )
rt_set_runnable_on_cpuid( &dynClampTask.rtTask, isolatedCPUId );
With that the RTAI loop runs on the first isolated CPU in case it was
specified via the kernel parameter isolcpus. Indeed, no need for RTAI to
provide any magic. Runs like a charm with the < 500ns jitter I reported in
my previous email.
Cheers,
Jan

Post by Paolo Mantegazza
You are right about the fact that there is no more the need to insmod
rtai_hal.ko stating the value of IsolCpuMask.
So I should have been clearer in my statement. The point is that it has
been left to give the possibility of checking isolcpus, as stated at boot,
with the expected one. Therefore, my suggestion was just my practice and
needs not to be followed.
As for making available the possibility of knowing the map of the cpus
isolated at boot, the matter is trivial.
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
#define CPU_ISOLATED_MAP (&cpu_isolated_map)
extern unsigned long cpu_isolated_map;
#else
#define CPU_ISOLATED_MAP (cpu_isolated_map)
extern cpumask_var_t cpu_isolated_map;
#endif
where "cpu_isolated_map" is exported from the kernel. It is simply the
boot isolcpus reflected in a bitmap.
So "cpu_isolated_map" is what you need to implement you task placing
policy using the available RTAI functions, we mentioned before.
In my view RTAI should only provide the tools to allow any user to
implement any allocation of the tasks on specific isolated cpus, especially
if they are more than one.
Paolo.

Post by Jan Benda
Hi Paolo,

First, let me say that it is likely you have a very good system. In fact
your performances without cpu isolation I somethin I would be dreaming for.
Even without creating any particular background Linux load, on the best
system I have, I get 3-4 times your values.
That said, CPU isolation, appropriately including the other two option
you mentions, is doing its duty.

Yes, I am actually thrilled by those latencies. CPU isolation on this
machine cuts the latencies by almost a factor two (both on an idle one and
under load).

1 - The kernel isolcpus are those that you should use for RTAI. So there
is nothing to else to guess and you should put your latency task there. Take
into account that rt_set_runnable_on_cpus will put your task on the least
used cpu within those you allowed. Naturally if the mask allows more than
one. If you have a single cpu of choice use rt_set_runnable_on_cpuid
instead. Moreover, you could use rt_task_init_cpuid, in place of
rt_task_init, and have your task placed on your cpu of choice from its very
beginning.

Yes, that I understand. What I want, however, that I isolate a CPU with
the kernelparameter, eg. isolcpus=2, and then have my kernel module
automatically select the isolated CPU for running the RTAI task.
When RTAI modules are loaded (without any parameter), then rtai_hal
notices the isolation mask from the linux kernel. Does this already
influence how RTAI tasks are handled? In particular, if rtai_hal knows that
CPU 2 is isolated, where will a rt_task_init job (without cpuid specified)
be executed, will it go on CPU 2? If yes then I am happy.
If no then, what I wanted to know is, how can I write some code in my
kernel module with an RTAI task, that reads the isolcpus mask, so that I can
pass it directly on to rt_set_runnable_on_cpus, or the corresponding
rt_task_init_cpu* function. Such that I do not have to change my code
when I play around with the isolcpus kernel parameter.

2) For sake of simplicity, assuming you have elected a single cpu (i.e.
rt_task_init_cpuid or rt_set_runnable_on_cpuid) you can copy the module parm
int mycpuid = 1; // default value
RTAI_MODULE_PARM(mycpuid, int);
MODULE_PARM_DESC(mycpuid, "The cpu number I want to run on (default: 1)")
.
.
rt_task_init_cpuid(%thread, fun, 0, 8192, 0, 1, 0, mycouid); // in
__latency_init

I wanted to suggest that somebody does this in the cvs...

It is also useful you to inform rtai_hal.ko thatbyou are isolating cpus
insmod /usr/realtime/modules/rtai_hal.ko IsolCpusMask=0x??;
where IsolCpusMask is the bit mask of your isolcpus.
In such a way also the Linux interrupt handlers will be removed from
your isolated cpus.

Now I am puzzled. In README.ISOLCPUS you say
"Starting from RTAI-4.1 rtai_hal.ko can inheredit the cpu isolation mask
from the kernel. So there is nomore the need to set IsolCpuMask at the
insmoding of rtai_hal.ko."
So if I isolate cpus using the isolcpus kernel parameter, there is no
reason to pass it again to rtai_hal via its IsolCpusMask, right?
Jan

Cheers, Paolo.

Post by Jan Benda
Hi,
recently I played around with the isolcpus kernel parameter (with great
1. Right now I select a CPU with the isolcpus kernel parameter and then
I also need to select the same CPU for the real time task / latency test.
Is there a way to figure out which CPUs are isolated and then use this
information for making the RTAI task run on that CPU (via
rt_set_runnable_on_cpus as in kern/latency test)?
2. I need to edit the latency-module.c file to change the CPU id on
which the test is run (modifying the RUNNABLE_ON_CPUS define). It would be
very nice if this would be a parameter that can be set when loading the
module - or as of 1) if the tests would be run on the isolated CPU. Same for
all the other tests in testsuite.
Here are the kern/latency tests with isolcpus=1 and latency test
Idle condition: average jitter (lat max - lat min) 250ns, maximum < 1000ns
Under full cpu, memory, io, and network load: average jitter 1200ns,
max about 2000ns
hal-linux-4.4.115-x86-10.patch. idle-poll and tsc-reliable.
Jan
_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

[Jan Benda]

Hi Wm,

sure I can give you this information.

I just created a repo on github where I uploaded my script that I use
for brute force checking of various kernel configurations and kernel
parameter:

https://github.com/relacs/makertai

There I also uploaded the test results together with information on the
machines of the two computers I am working with right now. Towards the
end of the latencies-* files (in the abbott* and mule* folders) you find
the output of 'lshw'. Is this enough information, or can you recommend
other ways on how to retrieve infos of interest from a computer?

I will add more data whenever I will test more machines. I would love if
other people could also provide their test results.

Regarding the kernel configuration: I only applied the minimum changes
that I have summarized at
http://relacs.sourceforge.net/plugins/rtaicomedi/index.html#basickernelconfiguration

In short, lshw says regarding the motherboard of my two machines:

abbott - with the awesome isolcpus performance (1 year old, from
https://www.csl-computer.com Aufrüst-PC 845 - Core i5-6600K):
description: Motherboard
product: Z170-K
vendor: ASUSTeK COMPUTER INC.
physical id: 0
version: Rev X.0x
serial: 161085640101302
slot: Default string
*-firmware
description: BIOS
vendor: American Megatrends Inc.
physical id: 0
version: 3301
date: 02/08/2017
size: 64KiB
capacity: 15MiB
capabilities: pci apm upgrade shadowing cdboot bootselect
socketedrom edd int13floppy1200 int13floppy720 int13floppy2880
int5printscreen int9keyboard int14serial int17printer acpi usb
biosbootspecification uefi

mule - the other also great machine (about 5(?) years old):
description: Mini Tower Computer
product: ESPRIMO P910 (S26361-Kxxx-Vyyy)
vendor: FUJITSU
version: C$PXW6
serial: YLJC035523
width: 64 bits
capabilities: smbios-2.7 dmi-2.7 vsyscall32
configuration: administrator_password=disabled boot=normal
chassis=mini-tower power-on_password=disabled sku=S26361-Kxxx-Vyyy
uuid=B8294DD0-F699-452D-BAE1-B8FC128054F6
*-core
description: Motherboard
product: D3162-A1
vendor: FUJITSU
physical id: 0
version: S26361-D3162-A1
serial: B22D7848
*-firmware
description: BIOS
vendor: FUJITSU // American Megatrends Inc.
physical id: 0
version: V4.6.5.3 R1.19.0 for D3162-A1x
date: 12/17/2012
size: 64KiB
capacity: 8128KiB
capabilities: pci upgrade shadowing cdboot bootselect edd
int5printscreen int9keyboard int14serial int17printer acpi usb
biosbootspecification uefi

I hope this helps!

Jan

Post by William Montgomery
Hi Jan,
The performance numbers you report are impressive.
Would you be able to share the specific motherboard or chipset used?
Regards,
Wm

Post by Jan Benda
Hi Paolo,
thank you for your hint on the cpu_isolated_mask. With that and some
grepping that lead me to the code for rtai_hal I came up with this solution
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
extern unsigned long cpu_isolated_map;
#else
extern cpumask_var_t cpu_isolated_map;
#endif
int isolatedCPUId = -1;
...
#ifdef CONFIG_SMP
for ( i=0; i<RTAI_NR_CPUS; i++ ) {
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
if ( cpu_isolated_map & (1<<i) ) {
#else
if ( cpumask_test_cpu( i, cpu_isolated_map ) ) {
#endif
isolatedCPUId = i;
break;
}
}
#endif
...
if ( isolatedCPUId >= 0 )
rt_set_runnable_on_cpuid( &dynClampTask.rtTask, isolatedCPUId );
With that the RTAI loop runs on the first isolated CPU in case it was
specified via the kernel parameter isolcpus. Indeed, no need for RTAI to
provide any magic. Runs like a charm with the < 500ns jitter I reported in
my previous email.
Cheers,
Jan

Post by Paolo Mantegazza
You are right about the fact that there is no more the need to insmod
rtai_hal.ko stating the value of IsolCpuMask.
So I should have been clearer in my statement. The point is that it has
been left to give the possibility of checking isolcpus, as stated at boot,
with the expected one. Therefore, my suggestion was just my practice and
needs not to be followed.
As for making available the possibility of knowing the map of the cpus
isolated at boot, the matter is trivial.
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
#define CPU_ISOLATED_MAP (&cpu_isolated_map)
extern unsigned long cpu_isolated_map;
#else
#define CPU_ISOLATED_MAP (cpu_isolated_map)
extern cpumask_var_t cpu_isolated_map;
#endif
where "cpu_isolated_map" is exported from the kernel. It is simply the
boot isolcpus reflected in a bitmap.
So "cpu_isolated_map" is what you need to implement you task placing
policy using the available RTAI functions, we mentioned before.
In my view RTAI should only provide the tools to allow any user to
implement any allocation of the tasks on specific isolated cpus, especially
if they are more than one.
Paolo.

Post by Jan Benda
Hi Paolo,

First, let me say that it is likely you have a very good system. In fact
your performances without cpu isolation I somethin I would be dreaming for.
Even without creating any particular background Linux load, on the best
system I have, I get 3-4 times your values.
That said, CPU isolation, appropriately including the other two option
you mentions, is doing its duty.

Yes, I am actually thrilled by those latencies. CPU isolation on this
machine cuts the latencies by almost a factor two (both on an idle one and
under load).

1 - The kernel isolcpus are those that you should use for RTAI. So there
is nothing to else to guess and you should put your latency task there. Take
into account that rt_set_runnable_on_cpus will put your task on the least
used cpu within those you allowed. Naturally if the mask allows more than
one. If you have a single cpu of choice use rt_set_runnable_on_cpuid
instead. Moreover, you could use rt_task_init_cpuid, in place of
rt_task_init, and have your task placed on your cpu of choice from its very
beginning.

Yes, that I understand. What I want, however, that I isolate a CPU with
the kernelparameter, eg. isolcpus=2, and then have my kernel module
automatically select the isolated CPU for running the RTAI task.
When RTAI modules are loaded (without any parameter), then rtai_hal
notices the isolation mask from the linux kernel. Does this already
influence how RTAI tasks are handled? In particular, if rtai_hal knows that
CPU 2 is isolated, where will a rt_task_init job (without cpuid specified)
be executed, will it go on CPU 2? If yes then I am happy.
If no then, what I wanted to know is, how can I write some code in my
kernel module with an RTAI task, that reads the isolcpus mask, so that I can
pass it directly on to rt_set_runnable_on_cpus, or the corresponding
rt_task_init_cpu* function. Such that I do not have to change my code
when I play around with the isolcpus kernel parameter.

2) For sake of simplicity, assuming you have elected a single cpu (i.e.
rt_task_init_cpuid or rt_set_runnable_on_cpuid) you can copy the module parm
int mycpuid = 1; // default value
RTAI_MODULE_PARM(mycpuid, int);
MODULE_PARM_DESC(mycpuid, "The cpu number I want to run on (default: 1)")
.
.
rt_task_init_cpuid(%thread, fun, 0, 8192, 0, 1, 0, mycouid); // in
__latency_init

I wanted to suggest that somebody does this in the cvs...

It is also useful you to inform rtai_hal.ko thatbyou are isolating cpus
insmod /usr/realtime/modules/rtai_hal.ko IsolCpusMask=0x??;
where IsolCpusMask is the bit mask of your isolcpus.
In such a way also the Linux interrupt handlers will be removed from
your isolated cpus.

Now I am puzzled. In README.ISOLCPUS you say
"Starting from RTAI-4.1 rtai_hal.ko can inheredit the cpu isolation mask
from the kernel. So there is nomore the need to set IsolCpuMask at the
insmoding of rtai_hal.ko."
So if I isolate cpus using the isolcpus kernel parameter, there is no
reason to pass it again to rtai_hal via its IsolCpusMask, right?
Jan

Cheers, Paolo.

Post by Jan Benda
Hi,
recently I played around with the isolcpus kernel parameter (with great
1. Right now I select a CPU with the isolcpus kernel parameter and then
I also need to select the same CPU for the real time task / latency test.
Is there a way to figure out which CPUs are isolated and then use this
information for making the RTAI task run on that CPU (via
rt_set_runnable_on_cpus as in kern/latency test)?
2. I need to edit the latency-module.c file to change the CPU id on
which the test is run (modifying the RUNNABLE_ON_CPUS define). It would be
very nice if this would be a parameter that can be set when loading the
module - or as of 1) if the tests would be run on the isolated CPU. Same for
all the other tests in testsuite.
Here are the kern/latency tests with isolcpus=1 and latency test
Idle condition: average jitter (lat max - lat min) 250ns, maximum < 1000ns
Under full cpu, memory, io, and network load: average jitter 1200ns,
max about 2000ns
hal-linux-4.4.115-x86-10.patch. idle-poll and tsc-reliable.
Jan
_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

[William Montgomery]

Hi Jan,

This is all the info I needed. Many thanks.

Regards,
Wm

Post by Jan Benda
Hi Wm,
sure I can give you this information.
I just created a repo on github where I uploaded my script that I use for
https://github.com/relacs/makertai
There I also uploaded the test results together with information on the
machines of the two computers I am working with right now. Towards the end
of the latencies-* files (in the abbott* and mule* folders) you find the
output of 'lshw'. Is this enough information, or can you recommend other
ways on how to retrieve infos of interest from a computer?
I will add more data whenever I will test more machines. I would love if
other people could also provide their test results.
Regarding the kernel configuration: I only applied the minimum changes that
I have summarized at
http://relacs.sourceforge.net/plugins/rtaicomedi/index.html#basickernelconfiguration
abbott - with the awesome isolcpus performance (1 year old, from
description: Motherboard
product: Z170-K
vendor: ASUSTeK COMPUTER INC.
physical id: 0
version: Rev X.0x
serial: 161085640101302
slot: Default string
*-firmware
description: BIOS
vendor: American Megatrends Inc.
physical id: 0
version: 3301
date: 02/08/2017
size: 64KiB
capacity: 15MiB
capabilities: pci apm upgrade shadowing cdboot bootselect
socketedrom edd int13floppy1200 int13floppy720 int13floppy2880
int5printscreen int9keyboard int14serial int17printer acpi usb
biosbootspecification uefi
description: Mini Tower Computer
product: ESPRIMO P910 (S26361-Kxxx-Vyyy)
vendor: FUJITSU
version: C$PXW6
serial: YLJC035523
width: 64 bits
capabilities: smbios-2.7 dmi-2.7 vsyscall32
configuration: administrator_password=disabled boot=normal
chassis=mini-tower power-on_password=disabled sku=S26361-Kxxx-Vyyy
uuid=B8294DD0-F699-452D-BAE1-B8FC128054F6
*-core
description: Motherboard
product: D3162-A1
vendor: FUJITSU
physical id: 0
version: S26361-D3162-A1
serial: B22D7848
*-firmware
description: BIOS
vendor: FUJITSU // American Megatrends Inc.
physical id: 0
version: V4.6.5.3 R1.19.0 for D3162-A1x
date: 12/17/2012
size: 64KiB
capacity: 8128KiB
capabilities: pci upgrade shadowing cdboot bootselect edd
int5printscreen int9keyboard int14serial int17printer acpi usb
biosbootspecification uefi
I hope this helps!
Jan

Post by William Montgomery
Hi Jan,
The performance numbers you report are impressive.
Would you be able to share the specific motherboard or chipset used?
Regards,
Wm

Post by Jan Benda
Hi Paolo,
thank you for your hint on the cpu_isolated_mask. With that and some
grepping that lead me to the code for rtai_hal I came up with this solution
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
extern unsigned long cpu_isolated_map;
#else
extern cpumask_var_t cpu_isolated_map;
#endif
int isolatedCPUId = -1;
...
#ifdef CONFIG_SMP
for ( i=0; i<RTAI_NR_CPUS; i++ ) {
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
if ( cpu_isolated_map & (1<<i) ) {
#else
if ( cpumask_test_cpu( i, cpu_isolated_map ) ) {
#endif
isolatedCPUId = i;
break;
}
}
#endif
...
if ( isolatedCPUId >= 0 )
rt_set_runnable_on_cpuid( &dynClampTask.rtTask, isolatedCPUId );
With that the RTAI loop runs on the first isolated CPU in case it was
specified via the kernel parameter isolcpus. Indeed, no need for RTAI to
provide any magic. Runs like a charm with the < 500ns jitter I reported in
my previous email.
Cheers,
Jan

Post by Paolo Mantegazza
You are right about the fact that there is no more the need to insmod
rtai_hal.ko stating the value of IsolCpuMask.
So I should have been clearer in my statement. The point is that it has
been left to give the possibility of checking isolcpus, as stated at boot,
with the expected one. Therefore, my suggestion was just my practice and
needs not to be followed.
As for making available the possibility of knowing the map of the cpus
isolated at boot, the matter is trivial.
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0)
#define CPU_ISOLATED_MAP (&cpu_isolated_map)
extern unsigned long cpu_isolated_map;
#else
#define CPU_ISOLATED_MAP (cpu_isolated_map)
extern cpumask_var_t cpu_isolated_map;
#endif
where "cpu_isolated_map" is exported from the kernel. It is simply the
boot isolcpus reflected in a bitmap.
So "cpu_isolated_map" is what you need to implement you task placing
policy using the available RTAI functions, we mentioned before.
In my view RTAI should only provide the tools to allow any user to
implement any allocation of the tasks on specific isolated cpus, especially
if they are more than one.
Paolo.

Post by Jan Benda
Hi Paolo,

First, let me say that it is likely you have a very good system. In fact
your performances without cpu isolation I somethin I would be dreaming for.
Even without creating any particular background Linux load, on the best
system I have, I get 3-4 times your values.
That said, CPU isolation, appropriately including the other two option
you mentions, is doing its duty.

Yes, I am actually thrilled by those latencies. CPU isolation on this
machine cuts the latencies by almost a factor two (both on an idle one and
under load).

1 - The kernel isolcpus are those that you should use for RTAI. So there
is nothing to else to guess and you should put your latency task there. Take
into account that rt_set_runnable_on_cpus will put your task on the least
used cpu within those you allowed. Naturally if the mask allows more than
one. If you have a single cpu of choice use rt_set_runnable_on_cpuid
instead. Moreover, you could use rt_task_init_cpuid, in place of
rt_task_init, and have your task placed on your cpu of choice from its very
beginning.

Yes, that I understand. What I want, however, that I isolate a CPU with
the kernelparameter, eg. isolcpus=2, and then have my kernel module
automatically select the isolated CPU for running the RTAI task.
When RTAI modules are loaded (without any parameter), then rtai_hal
notices the isolation mask from the linux kernel. Does this already
influence how RTAI tasks are handled? In particular, if rtai_hal knows that
CPU 2 is isolated, where will a rt_task_init job (without cpuid specified)
be executed, will it go on CPU 2? If yes then I am happy.
If no then, what I wanted to know is, how can I write some code in my
kernel module with an RTAI task, that reads the isolcpus mask, so that I can
pass it directly on to rt_set_runnable_on_cpus, or the corresponding
rt_task_init_cpu* function. Such that I do not have to change my code
when I play around with the isolcpus kernel parameter.

2) For sake of simplicity, assuming you have elected a single cpu (i.e.
rt_task_init_cpuid or rt_set_runnable_on_cpuid) you can copy the module parm
int mycpuid = 1; // default value
RTAI_MODULE_PARM(mycpuid, int);
MODULE_PARM_DESC(mycpuid, "The cpu number I want to run on (default: 1)")
.
.
rt_task_init_cpuid(%thread, fun, 0, 8192, 0, 1, 0, mycouid); // in
__latency_init

I wanted to suggest that somebody does this in the cvs...

It is also useful you to inform rtai_hal.ko thatbyou are isolating cpus
insmod /usr/realtime/modules/rtai_hal.ko IsolCpusMask=0x??;
where IsolCpusMask is the bit mask of your isolcpus.
In such a way also the Linux interrupt handlers will be removed from
your isolated cpus.

Now I am puzzled. In README.ISOLCPUS you say
"Starting from RTAI-4.1 rtai_hal.ko can inheredit the cpu isolation mask
from the kernel. So there is nomore the need to set IsolCpuMask at the
insmoding of rtai_hal.ko."
So if I isolate cpus using the isolcpus kernel parameter, there is no
reason to pass it again to rtai_hal via its IsolCpusMask, right?
Jan

Cheers, Paolo.

Post by Jan Benda
Hi,
recently I played around with the isolcpus kernel parameter (with great
1. Right now I select a CPU with the isolcpus kernel parameter and then
I also need to select the same CPU for the real time task / latency test.
Is there a way to figure out which CPUs are isolated and then use this
information for making the RTAI task run on that CPU (via
rt_set_runnable_on_cpus as in kern/latency test)?
2. I need to edit the latency-module.c file to change the CPU id on
which the test is run (modifying the RUNNABLE_ON_CPUS define). It would be
very nice if this would be a parameter that can be set when loading the
module - or as of 1) if the tests would be run on the isolated CPU. Same for
all the other tests in testsuite.
Here are the kern/latency tests with isolcpus=1 and latency test
Idle condition: average jitter (lat max - lat min) 250ns, maximum < 1000ns
Under full cpu, memory, io, and network load: average jitter 1200ns,
max about 2000ns
hal-linux-4.4.115-x86-10.patch. idle-poll and tsc-reliable.
Jan
_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai

_______________________________________________
Rtai mailing list
https://mail.rtai.org/cgi-bin/mailman/listinfo/rtai