icache-dcache coherence on ARM

icache-dcache coherence on ARM

[Xiaozhu Meng]

Hi,

I am reading gdb's source code to hopefully get answers for a question that
I have in my other project.

On ARM, the architecture does not guarantee that icache and dcache are
coherent. When GDB writes a software breakpoint into the inferior's address
space, is it possible that the inferior executes outdated code in icache
and thus miss the software breakpoint?

I try to search around the gdb code base to understand whether GDB flushes
icache or not, but could not find answers.

I appreciate any feedback!

Thanks,

--Xiaozhu

Re: icache-dcache coherence on ARM

[John Baldwin]

On 5/6/19 12:30 PM, Xiaozhu Meng wrote:
> Hi,
> 
> I am reading gdb's source code to hopefully get answers for a question that
> I have in my other project.
> 
> On ARM, the architecture does not guarantee that icache and dcache are
> coherent. When GDB writes a software breakpoint into the inferior's address
> space, is it possible that the inferior executes outdated code in icache
> and thus miss the software breakpoint?
> 
> I try to search around the gdb code base to understand whether GDB flushes
> icache or not, but could not find answers.
> 
> I appreciate any feedback!

I suspect that the cache flushing is done by the host OS kernel in response
to the write.  This is what happens on FreeBSD at least where any executable
page in a process written to via ptrace(PT_IO) has its i-cache flushed by this
code in sys/kern/sys_process.c in proc_rwmem():

                /*
                 * Now do the i/o move.
                 */
                error = uiomove_fromphys(&m, page_offset, len, uio);

                /* Make the I-cache coherent for breakpoints. */
                if (writing && error == 0) {
                        vm_map_lock_read(map);
                        if (vm_map_check_protection(map, pageno, pageno +
                            PAGE_SIZE, VM_PROT_EXECUTE))
                                vm_sync_icache(map, uva, len);
                        vm_map_unlock_read(map);
                }


-- 
John Baldwin

Re: icache-dcache coherence on ARM

[Xiaozhu Meng]

Hi John,

Thanks for your reply!

I asked this question because our project on Linux actually encountered
this problem where we use ptrace to write new code into the inferior and
then continue the inferior. The continued inferior sometimes works as
expected, but sometimes crashes due to SIGILLs on seemingly legitimate
instructions.

So, I am very interested in seeing how GDB deals with this problem on
Linux.

Thanks,

--Xiaozhu

On Mon, May 6, 2019 at 3:52 PM John Baldwin <jhb@freebsd.org> wrote:

> On 5/6/19 12:30 PM, Xiaozhu Meng wrote:
> > Hi,
> >
> > I am reading gdb's source code to hopefully get answers for a question
> that
> > I have in my other project.
> >
> > On ARM, the architecture does not guarantee that icache and dcache are
> > coherent. When GDB writes a software breakpoint into the inferior's
> address
> > space, is it possible that the inferior executes outdated code in icache
> > and thus miss the software breakpoint?
> >
> > I try to search around the gdb code base to understand whether GDB
> flushes
> > icache or not, but could not find answers.
> >
> > I appreciate any feedback!
>
> I suspect that the cache flushing is done by the host OS kernel in response
> to the write.  This is what happens on FreeBSD at least where any
> executable
> page in a process written to via ptrace(PT_IO) has its i-cache flushed by
> this
> code in sys/kern/sys_process.c in proc_rwmem():
>
>                 /*
>                  * Now do the i/o move.
>                  */
>                 error = uiomove_fromphys(&m, page_offset, len, uio);
>
>                 /* Make the I-cache coherent for breakpoints. */
>                 if (writing && error == 0) {
>                         vm_map_lock_read(map);
>                         if (vm_map_check_protection(map, pageno, pageno +
>                             PAGE_SIZE, VM_PROT_EXECUTE))
>                                 vm_sync_icache(map, uva, len);
>                         vm_map_unlock_read(map);
>                 }
>
>
> --
> John Baldwin
>

Re: icache-dcache coherence on ARM

[John Baldwin]

On 5/6/19 2:17 PM, Xiaozhu Meng wrote:
> Hi John,
> 
> Thanks for your reply!
> 
> I asked this question because our project on Linux actually encountered
> this problem where we use ptrace to write new code into the inferior and
> then continue the inferior. The continued inferior sometimes works as
> expected, but sometimes crashes due to SIGILLs on seemingly legitimate
> instructions.
> 
> So, I am very interested in seeing how GDB deals with this problem on
> Linux.

I do not see any explicit cache management in linux-nat.c or arm-linux-nat.c,
so my best guess is that GDB is relying on the kernel to manage this on
Linux as well.  I do see that TARGET_OBJECT_MEMORY on Linux can sometimes
use /proc/<pid>/mem instead of ptrace().  I'm not very familiar with the
Linux kernel, but one thing to check might be that both ptrace and procfs
are doing the i-cache invalidation.

-- 
John Baldwin

Re: icache-dcache coherence on ARM

[Xiaozhu Meng]

Hi John,

Thanks for hinting that GDB may also use procfs. I replaced ptrace() with
procfs to write code to the inferior and my code is now always working as
expected!

I appreciate your helps!

Thanks,

--Xiaozhu

On Mon, May 6, 2019 at 4:37 PM John Baldwin <jhb@freebsd.org> wrote:

> On 5/6/19 2:17 PM, Xiaozhu Meng wrote:
> > Hi John,
> >
> > Thanks for your reply!
> >
> > I asked this question because our project on Linux actually encountered
> > this problem where we use ptrace to write new code into the inferior and
> > then continue the inferior. The continued inferior sometimes works as
> > expected, but sometimes crashes due to SIGILLs on seemingly legitimate
> > instructions.
> >
> > So, I am very interested in seeing how GDB deals with this problem on
> > Linux.
>
> I do not see any explicit cache management in linux-nat.c or
> arm-linux-nat.c,
> so my best guess is that GDB is relying on the kernel to manage this on
> Linux as well.  I do see that TARGET_OBJECT_MEMORY on Linux can sometimes
> use /proc/<pid>/mem instead of ptrace().  I'm not very familiar with the
> Linux kernel, but one thing to check might be that both ptrace and procfs
> are doing the i-cache invalidation.
>
> --
> John Baldwin
>