Discussion:
[Rtai] RTAI-5.1 "insmod rtai_sched.ko" lockup on Core2 Duo platform
Vladimir Cotfas
2018-08-17 21:18:18 UTC
Permalink
Hi,

I tried compiling 4.9.80 as SMP and UP and in both cases Linux locks up
when I load rtai_sched.ko (in the SMP case insmod is blocked in a read of
the file rtai_sched.ko).

I attached my UP config file for the Core2Duo.

I compiled RTAI-5.1 like so:

./configure --disable-netrpc --enable-sem --enable-mbx --enable-rtdm
--enable-rtdm-select --enable-fifos --enable-mbx --enable-shm=m
--with-linux-dir=$HOME/linux-4.9.80 --prefix=$HOME/realtime


Linux complains about a hard lockup in start_kernel->rest_init as seen in
the screen cap [https://app.box.com/s/hb1qya3u3asbv06y7yk2nzhuaq8htteh].
Cap also shows the insmod sequence.

Has anybody seen something similar?

Thanks, Vlad
Paolo Mantegazza
2018-08-18 08:23:40 UTC
Permalink
It is likely related to the latency calibration, carried out at the very first use of rtai_sched only.
AFAIK, when it happens is related to the Linux config.
Have you cared of README.CONF_RMRKS?
In any case you can check if it so by bypassing the calibration, either through, e.g.:
insmod rtai_sched.ko kernel_latency= 700 user_latency=1000
or by creating a single line file "latencies", in the directory "calibration" of the RTAI install dir, which should read:
700 1000 100000
There are cases reported in the RTAI list archives.
Let me know.

Paolo
________________________________________
From: Rtai [rtai-***@rtai.org] on behalf of Vladimir Cotfas [***@mantatest.com]
Sent: Friday, August 17, 2018 11:18 PM
To: rtai-***@rtai.org; ***@rtai.org; Scott Gilbertson
Subject: [Rtai] RTAI-5.1 "insmod rtai_sched.ko" lockup on Core2 Duo platform

Hi,

I tried compiling 4.9.80 as SMP and UP and in both cases Linux locks up when I load rtai_sched.ko (in the SMP case insmod is blocked in a read of the file rtai_sched.ko).

I attached my UP config file for the Core2Duo.

I compiled RTAI-5.1 like so:
./configure --disable-netrpc --enable-sem --enable-mbx --enable-rtdm --enable-rtdm-select --enable-fifos --enable-mbx --enable-shm=m --with-linux-dir=$HOME/linux-4.9.80 --prefix=$HOME/realtime

Linux complains about a hard lockup in start_kernel->rest_init as seen in the screen cap [https://app.box.com/s/hb1qya3u3asbv06y7yk2nzhuaq8htteh]. Cap also shows the insmod sequence.

Has anybody seen something similar?

Thanks, Vlad
Vladimir Cotfas
2018-08-20 19:04:11 UTC
Permalink
Paolo,

Yes I went thru that README and a few of your mailing list posts.

With your params it still gets stuck on loading rtai_sched.ko -- [
https://app.box.com/s/55nv3ryar95s3wbiyba6paygjo2829o9]

I have done "strace /sbin/insmpd rtai_scked.ko" and it gets stuck in a
read() *before* init_module("ELF", 0xNNNN) which never happens.

I observed that just by loading rtai_hal.ko the box becomes very sluggish
afterwards and subsequent loading of other RTAI ko files gets stuck in
exactly the same read() so I don't think the code in rtai_sched.ko ever
gets executed.

It appears it is enough to load rtai_hal.ko (no neeed for rtai_sched.ko) to
trigger the lockup:

[ 31.806592] rtai_hal: loading out-of-tree module taints kernel.
[ 31.806791] I-pipe: head domain RTAI registered.
[ 31.806792] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0.
[ 31.806795] SYSINFO - # CPUs: 1, TIMER NAME: 'lapic', TIMER IRQ: 2306,
TIMER FREQ: 10418019, CLOCK NAME: 'tsc', CLOCK FREQ: 2000163000, CPU FREQ:
2000163000, LINUX TIMER IRQ: 2306.
[ 31.806796]
[ 31.806796] REMARK: RTAI WILL NOT ACCESS USER SPACE ON STACKS ARGS ITS
WAY.
[ 31.806796]
[ 31.814115] EXT4-fs (hda1): re-mounted. Opts: commit=20
[ 31.814115] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0[
31.814115] CPU: 0 PID: 0 Comm: swapper Tainted: G O
4.9.80-ipipe #11
[ 31.814115] Hardware name: /945GM, BIOS 6.00 PG 12/23/2009
[ 31.814115] task: c15cbd80 task.stack: c15c6000
[ 31.814115] Stack:
[ 31.814115] c141407f c15c7fc0 c1059a99 c15cbd80 c15cbd80 c166d800
c15cbd80 c15cbd80
[ 31.814115] c163f0e0 0009f600 c166d800 c15c7fc8 c14102b7 c15c7fec
c160abc3 000000bb
[ 31.814115] ffffffff ffffffff 00000000 c160a64b c163f0e0 00020800
c15c7ff8 c160a34d
[ 31.814115] Call Trace:
[ 31.814115] [<c141407f>] ? default_idle_call+0x2f/0x40
[ 31.814115] [<c1059a99>] cpu_startup_entry+0xa9/0x1b0
[ 31.814115] [<c14102b7>] rest_init+0x57/0x60
[ 31.814115] [<c160abc3>] start_kernel+0x38e/0x394
[ 31.814115] [<c160a64b>] ? set_init_arg+0x52/0x52
[ 31.814115] [<c160a34d>] i386_start_kernel+0x138/0x13b
[ 31.814115] Code: 08 e8 35 8b fb ff fa eb 99 c1 e3 06 89 d9 8b 93 8c bd
70 c1 81 c1 80 bd 70 c1 ff 51 08 fa eb 82 8d 74 26 00 8d bc 27 00 00 00 00
<55> 89 e5 fa 0f ba 35 20 f5 5d c1 00 a1 24 f5 5d c1 85 c0 75 03


Is there a way to trace rtai_hal.ko?

Thanks, Vlad

On Sat, Aug 18, 2018 at 4:23 AM, Paolo Mantegazza <
***@polimi.it> wrote:

> It is likely related to the latency calibration, carried out at the very
> first use of rtai_sched only.
> AFAIK, when it happens is related to the Linux config.
> Have you cared of README.CONF_RMRKS?
> In any case you can check if it so by bypassing the calibration, either
> through, e.g.:
> insmod rtai_sched.ko kernel_latency= 700 user_latency=1000
> or by creating a single line file "latencies", in the directory
> "calibration" of the RTAI install dir, which should read:
> 700 1000 100000
> There are cases reported in the RTAI list archives.
> Let me know.
>
> Paolo
> ________________________________________
> From: Rtai [rtai-***@rtai.org] on behalf of Vladimir Cotfas [
> ***@mantatest.com]
> Sent: Friday, August 17, 2018 11:18 PM
> To: rtai-***@rtai.org; ***@rtai.org; Scott Gilbertson
> Subject: [Rtai] RTAI-5.1 "insmod rtai_sched.ko" lockup on Core2 Duo
> platform
>
> Hi,
>
> I tried compiling 4.9.80 as SMP and UP and in both cases Linux locks up
> when I load rtai_sched.ko (in the SMP case insmod is blocked in a read of
> the file rtai_sched.ko).
>
> I attached my UP config file for the Core2Duo.
>
> I compiled RTAI-5.1 like so:
> ./configure --disable-netrpc --enable-sem --enable-mbx --enable-rtdm
> --enable-rtdm-select --enable-fifos --enable-mbx --enable-shm=m
> --with-linux-dir=$HOME/linux-4.9.80 --prefix=$HOME/realtime
>
> Linux complains about a hard lockup in start_kernel->rest_init as seen in
> the screen cap [https://app.box.com/s/hb1qya3u3asbv06y7yk2nzhuaq8htteh].
> Cap also shows the insmod sequence.
>
> Has anybody seen something similar?
>
> Thanks, Vlad
>
>
>
>
>
>
Vladimir Cotfas
2018-08-20 23:17:45 UTC
Permalink
Paolo,

I corrected the CPU mismatch (attached) but no change. My dev VM is
Slackware 14.1 32 bit with gcc version 4.8.2 (GCC).

Please do send me your .config file for 4.9.51 and I'll do some diffing and
try to apply it to my Core2 Duo as 32 bit and UP then SMP.

I looked a bit at hal.c and it seems to forward calls to some ipipe_()
functions.

So the question for adding traces is W*hat is rtai_hal.ko supposed to do
all by itself?* (with no other RTAI bits loaded).

Previously I had kernel 2.6.29 and RTAI 3.7.1 working well:

I-pipe: Domain RTAI registered.
RTAI[hal]: <3.7.1> mounted over IPIPE-NOTHREADS 2.4-01.
RTAI[hal]: compiled with gcc version 4.3.4 (GCC) .
RTAI[hal]: mounted (IPIPE-NOTHREADS, IMMEDIATE (INTERNAL IRQs DISPATCHED),
ISOL_CPUS_MASK: 0).
PIPELINE layers:
f80615c0 9ac15d93 RTAI 200
c0446ac0 0 Linux 100


Thanks, Vlad

On Mon, Aug 20, 2018 at 5:17 PM, Paolo Mantegazza <
***@polimi.it> wrote:

> Just to be sure, since it is a bit of time since I tested 5.1 UP, I've
> checked rtai-5.1 UP, with linux 4.9.51, because I had installed already.
> I do not expect any difference using 4.9.80.
> My machine is 64 bits. So it does not match you 32 bits one.
> Unfortunately, I will have no access to my old PIII, 32 bits, till late
> next week for a further UP check, closer to your arch.
>
> I suspect it could be a matter of Linux config. Is not there a clash in
> having CONFIG_MCORE2=y and CONFIG_X86_GENERIC=y set together?
> If you want I can pass to you the 64 bits Linux . config, I have just used
> for my 64 bits UP.
>
> As for debugging rtai_hal.ko, you can use printk to track its insmod. Then
> you could place othe printks in the irq dispatcher, function
> rtai_hirq_dispatcher.
>
> Paolo.
>
> From: Vladimir Cotfas [***@mantatest.com]
> Sent: Monday, August 20, 2018 9:04 PM
> To: Paolo Mantegazza
> Cc: rtai-***@rtai.org; ***@rtai.org; Scott Gilbertson
> Subject: Re: [Rtai] RTAI-5.1 "insmod rtai_sched.ko" lockup on Core2 Duo
> platform
>
> Paolo,
>
> Yes I went thru that README and a few of your mailing list posts.
>
> With your params it still gets stuck on loading rtai_sched.ko -- [
> https://app.box.com/s/55nv3ryar95s3wbiyba6paygjo2829o9]
>
> I have done "strace /sbin/insmpd rtai_scked.ko" and it gets stuck in a
> read() before init_module("ELF", 0xNNNN) which never happens.
>
> I observed that just by loading rtai_hal.ko the box becomes very sluggish
> afterwards and subsequent loading of other RTAI ko files gets stuck in
> exactly the same read() so I don't think the code in rtai_sched.ko ever
> gets executed.
>
> It appears it is enough to load rtai_hal.ko (no neeed for rtai_sched.ko)
> to trigger the lockup:
>
> [ 31.806592] rtai_hal: loading out-of-tree module taints kernel.
> [ 31.806791] I-pipe: head domain RTAI registered.
> [ 31.806792] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0.
> [ 31.806795] SYSINFO - # CPUs: 1, TIMER NAME: 'lapic', TIMER IRQ: 2306,
> TIMER FREQ: 10418019, CLOCK NAME: 'tsc', CLOCK FREQ: 2000163000, CPU FREQ:
> 2000163000, LINUX TIMER IRQ: 2306.
> [ 31.806796]
> [ 31.806796] REMARK: RTAI WILL NOT ACCESS USER SPACE ON STACKS ARGS ITS
> WAY.
> [ 31.806796]
> [ 31.814115] EXT4-fs (hda1): re-mounted. Opts: commit=20
> [ 31.814115] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0[
> 31.814115] CPU: 0 PID: 0 Comm: swapper Tainted: G O
> 4.9.80-ipipe #11
> [ 31.814115] Hardware name: /945GM, BIOS 6.00 PG 12/23/2009
> [ 31.814115] task: c15cbd80 task.stack: c15c6000
> [ 31.814115] Stack:
> [ 31.814115] c141407f c15c7fc0 c1059a99 c15cbd80 c15cbd80 c166d800
> c15cbd80 c15cbd80
> [ 31.814115] c163f0e0 0009f600 c166d800 c15c7fc8 c14102b7 c15c7fec
> c160abc3 000000bb
> [ 31.814115] ffffffff ffffffff 00000000 c160a64b c163f0e0 00020800
> c15c7ff8 c160a34d
> [ 31.814115] Call Trace:
> [ 31.814115] [<c141407f>] ? default_idle_call+0x2f/0x40
> [ 31.814115] [<c1059a99>] cpu_startup_entry+0xa9/0x1b0
> [ 31.814115] [<c14102b7>] rest_init+0x57/0x60
> [ 31.814115] [<c160abc3>] start_kernel+0x38e/0x394
> [ 31.814115] [<c160a64b>] ? set_init_arg+0x52/0x52
> [ 31.814115] [<c160a34d>] i386_start_kernel+0x138/0x13b
> [ 31.814115] Code: 08 e8 35 8b fb ff fa eb 99 c1 e3 06 89 d9 8b 93 8c bd
> 70 c1 81 c1 80 bd 70 c1 ff 51 08 fa eb 82 8d 74 26 00 8d bc 27 00 00 00 00
> <55> 89 e5 fa 0f ba 35 20 f5 5d c1 00 a1 24 f5 5d c1 85 c0 75 03
>
> Is there a way to trace rtai_hal.ko?
>
> Thanks, Vlad
>
> On Sat, Aug 18, 2018 at 4:23 AM, Paolo Mantegazza <
> ***@polimi.it<mailto:***@polimi.it>> wrote:
> It is likely related to the latency calibration, carried out at the very
> first use of rtai_sched only.
> AFAIK, when it happens is related to the Linux config.
> Have you cared of README.CONF_RMRKS?
> In any case you can check if it so by bypassing the calibration, either
> through, e.g.:
> insmod rtai_sched.ko kernel_latency= 700 user_latency=1000
> or by creating a single line file "latencies", in the directory
> "calibration" of the RTAI install dir, which should read:
> 700 1000 100000
> There are cases reported in the RTAI list archives.
> Let me know.
>
> Paolo
> ________________________________________
> From: Rtai [rtai-***@rtai.org<mailto:rtai-***@rtai.org>] on
> behalf of Vladimir Cotfas [***@mantatest.com<mailto:***@mantatest.com
> >]
> Sent: Friday, August 17, 2018 11:18 PM
> To: rtai-***@rtai.org<mailto:rtai-***@rtai.org>; ***@rtai.org
> <mailto:***@rtai.org>; Scott Gilbertson
> Subject: [Rtai] RTAI-5.1 "insmod rtai_sched.ko" lockup on Core2 Duo
> platform
>
> Hi,
>
> I tried compiling 4.9.80 as SMP and UP and in both cases Linux locks up
> when I load rtai_sched.ko (in the SMP case insmod is blocked in a read of
> the file rtai_sched.ko).
>
> I attached my UP config file for the Core2Duo.
>
> I compiled RTAI-5.1 like so:
> ./configure --disable-netrpc --enable-sem --enable-mbx --enable-rtdm
> --enable-rtdm-select --enable-fifos --enable-mbx --enable-shm=m
> --with-linux-dir=$HOME/linux-4.9.80 --prefix=$HOME/realtime
>
> Linux complains about a hard lockup in start_kernel->rest_init as seen in
> the screen cap [https://app.box.com/s/hb1qya3u3asbv06y7yk2nzhuaq8htteh].
> Cap also shows the insmod sequence.
>
> Has anybody seen something similar?
>
> Thanks, Vlad
>
>
>
>
>
>
>
Paolo Mantegazza
2018-08-22 19:53:39 UTC
Permalink
Rtai_hal.ko alone will just trap Linux irqs and pend them to be processed when there is no real time activity, which is the only thing it has to do in your case. Not always so though, as with rtai_hal it is possible to do a few interesting things. Which, I guess, is not your case.

I've attached mu Linux .config.

Paolo
________________________________________
From: Vladimir Cotfas [***@mantatest.com]
Sent: Tuesday, August 21, 2018 1:17 AM
To: Paolo Mantegazza
Cc: rtai-***@rtai.org; ***@rtai.org; Scott Gilbertson
Subject: Re: [Rtai] RTAI-5.1 "insmod rtai_sched.ko" lockup on Core2 Duo platform

Paolo,

I corrected the CPU mismatch (attached) but no change. My dev VM is Slackware 14.1 32 bit with gcc version 4.8.2 (GCC).

Please do send me your .config file for 4.9.51 and I'll do some diffing and try to apply it to my Core2 Duo as 32 bit and UP then SMP.

I looked a bit at hal.c and it seems to forward calls to some ipipe_() functions.

So the question for adding traces is What is rtai_hal.ko supposed to do all by itself? (with no other RTAI bits loaded).

Previously I had kernel 2.6.29 and RTAI 3.7.1 working well:
I-pipe: Domain RTAI registered.
RTAI[hal]: <3.7.1> mounted over IPIPE-NOTHREADS 2.4-01.
RTAI[hal]: compiled with gcc version 4.3.4 (GCC) .
RTAI[hal]: mounted (IPIPE-NOTHREADS, IMMEDIATE (INTERNAL IRQs DISPATCHED), ISOL_CPUS_MASK: 0).
PIPELINE layers:
f80615c0 9ac15d93 RTAI 200
c0446ac0 0 Linux 100

Thanks, Vlad

On Mon, Aug 20, 2018 at 5:17 PM, Paolo Mantegazza <***@polimi.it<mailto:***@polimi.it>> wrote:
Just to be sure, since it is a bit of time since I tested 5.1 UP, I've checked rtai-5.1 UP, with linux 4.9.51, because I had installed already.
I do not expect any difference using 4.9.80.
My machine is 64 bits. So it does not match you 32 bits one. Unfortunately, I will have no access to my old PIII, 32 bits, till late next week for a further UP check, closer to your arch.

I suspect it could be a matter of Linux config. Is not there a clash in having CONFIG_MCORE2=y and CONFIG_X86_GENERIC=y set together?
If you want I can pass to you the 64 bits Linux . config, I have just used for my 64 bits UP.

As for debugging rtai_hal.ko, you can use printk to track its insmod. Then you could place othe printks in the irq dispatcher, function rtai_hirq_dispatcher.

Paolo.

From: Vladimir Cotfas [***@mantatest.com<mailto:***@mantatest.com>]
Sent: Monday, August 20, 2018 9:04 PM
To: Paolo Mantegazza
Cc: rtai-***@rtai.org<mailto:rtai-***@rtai.org>; ***@rtai.org<mailto:***@rtai.org>; Scott Gilbertson
Subject: Re: [Rtai] RTAI-5.1 "insmod rtai_sched.ko" lockup on Core2 Duo platform

Paolo,

Yes I went thru that README and a few of your mailing list posts.

With your params it still gets stuck on loading rtai_sched.ko -- [https://app.box.com/s/55nv3ryar95s3wbiyba6paygjo2829o9]

I have done "strace /sbin/insmpd rtai_scked.ko" and it gets stuck in a read() before init_module("ELF", 0xNNNN) which never happens.

I observed that just by loading rtai_hal.ko the box becomes very sluggish afterwards and subsequent loading of other RTAI ko files gets stuck in exactly the same read() so I don't think the code in rtai_sched.ko ever gets executed.

It appears it is enough to load rtai_hal.ko (no neeed for rtai_sched.ko) to trigger the lockup:

[ 31.806592] rtai_hal: loading out-of-tree module taints kernel.
[ 31.806791] I-pipe: head domain RTAI registered.
[ 31.806792] RTAI[hal]: mounted. ISOL_CPUS_MASK: 0.
[ 31.806795] SYSINFO - # CPUs: 1, TIMER NAME: 'lapic', TIMER IRQ: 2306, TIMER FREQ: 10418019, CLOCK NAME: 'tsc', CLOCK FREQ: 2000163000, CPU FREQ: 2000163000, LINUX TIMER IRQ: 2306.
[ 31.806796]
[ 31.806796] REMARK: RTAI WILL NOT ACCESS USER SPACE ON STACKS ARGS ITS WAY.
[ 31.806796]
[ 31.814115] EXT4-fs (hda1): re-mounted. Opts: commit=20
[ 31.814115] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0[ 31.814115] CPU: 0 PID: 0 Comm: swapper Tainted: G O 4.9.80-ipipe #11
[ 31.814115] Hardware name: /945GM, BIOS 6.00 PG 12/23/2009
[ 31.814115] task: c15cbd80 task.stack: c15c6000
[ 31.814115] Stack:
[ 31.814115] c141407f c15c7fc0 c1059a99 c15cbd80 c15cbd80 c166d800 c15cbd80 c15cbd80
[ 31.814115] c163f0e0 0009f600 c166d800 c15c7fc8 c14102b7 c15c7fec c160abc3 000000bb
[ 31.814115] ffffffff ffffffff 00000000 c160a64b c163f0e0 00020800 c15c7ff8 c160a34d
[ 31.814115] Call Trace:
[ 31.814115] [<c141407f>] ? default_idle_call+0x2f/0x40
[ 31.814115] [<c1059a99>] cpu_startup_entry+0xa9/0x1b0
[ 31.814115] [<c14102b7>] rest_init+0x57/0x60
[ 31.814115] [<c160abc3>] start_kernel+0x38e/0x394
[ 31.814115] [<c160a64b>] ? set_init_arg+0x52/0x52
[ 31.814115] [<c160a34d>] i386_start_kernel+0x138/0x13b
[ 31.814115] Code: 08 e8 35 8b fb ff fa eb 99 c1 e3 06 89 d9 8b 93 8c bd 70 c1 81 c1 80 bd 70 c1 ff 51 08 fa eb 82 8d 74 26 00 8d bc 27 00 00 00 00 <55> 89 e5 fa 0f ba 35 20 f5 5d c1 00 a1 24 f5 5d c1 85 c0 75 03

Is there a way to trace rtai_hal.ko?

Thanks, Vlad

On Sat, Aug 18, 2018 at 4:23 AM, Paolo Mantegazza <***@polimi.it<mailto:***@polimi.it><mailto:***@polimi.it<mailto:***@polimi.it>>> wrote:
It is likely related to the latency calibration, carried out at the very first use of rtai_sched only.
AFAIK, when it happens is related to the Linux config.
Have you cared of README.CONF_RMRKS?
In any case you can check if it so by bypassing the calibration, either through, e.g.:
insmod rtai_sched.ko kernel_latency= 700 user_latency=1000
or by creating a single line file "latencies", in the directory "calibration" of the RTAI install dir, which should read:
700 1000 100000
There are cases reported in the RTAI list archives.
Let me know.

Paolo
________________________________________
From: Rtai [rtai-***@rtai.org<mailto:rtai-***@rtai.org><mailto:rtai-***@rtai.org<mailto:rtai-***@rtai.org>>] on behalf of Vladimir Cotfas [***@mantatest.com<mailto:***@mantatest.com><mailto:***@mantatest.com<mailto:***@mantatest.com>>]
Sent: Friday, August 17, 2018 11:18 PM
To: rtai-***@rtai.org<mailto:rtai-***@rtai.org><mailto:rtai-***@rtai.org<mailto:rtai-***@rtai.org>>; ***@rtai.org<mailto:***@rtai.org><mailto:***@rtai.org<mailto:***@rtai.org>>; Scott Gilbertson
Subject: [Rtai] RTAI-5.1 "insmod rtai_sched.ko" lockup on Core2 Duo platform

Hi,

I tried compiling 4.9.80 as SMP and UP and in both cases Linux locks up when I load rtai_sched.ko (in the SMP case insmod is blocked in a read of the file rtai_sched.ko).

I attached my UP config file for the Core2Duo.

I compiled RTAI-5.1 like so:
./configure --disable-netrpc --enable-sem --enable-mbx --enable-rtdm --enable-rtdm-select --enable-fifos --enable-mbx --enable-shm=m --with-linux-dir=$HOME/linux-4.9.80 --prefix=$HOME/realtime

Linux complains about a hard lockup in start_kernel->rest_init as seen in the screen cap [https://app.box.com/s/hb1qya3u3asbv06y7yk2nzhuaq8htteh]. Cap also shows the insmod sequence.

Has anybody seen something similar?

Thanks, Vlad
Loading...