[RFC] Enable GDB handle compressed target.xml returned by GDB stub

[RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

Hi,

As stated in GDB document, the most natural way for gdb to get target
register information is the target.xml file returned from gdb server
or gdb stub. Normally it wouldn't be a burden for gdb server/stub to
implement such feature, but there are some gdb stubs running directly
on bare metal boards which have very limited memory/flash resources,
saving the content of target.xml file means extra storage and cost
increment. So a practical solution for such stub is to use the content
of pre-compressed target.xml file and reply host gdb with that
content. To cope with such compressed xml file, I am going to propose
following working model:
1). Use ZLIB to compress (at Z_BEST_COMPRESSION level) and decompress
the xml file in memory, don't use the gz format.
2). When reply to host gdb, the first four bytes of the packet should
be "ZLIB" and following four bytes should be the length of data before
compressed, the rest should be the compressed data. In this way, the
host gdb can know the format of received data and how to decompress
them.

I am not going to propose to adding a new feature in gdb remote serial
protocol because it is possible that a gdb server/stub can support
both ways. In addition gdb needs to know the length of uncompressed
data. Anyway we need to detect the received data to figure out its
actual format. So I plan to implement such functionality as an
extension of how gdb handle received xml file. Perhaps the right place
for this functionality is right after the call of function
fetch_available_features_from_target in function
target_read_description_xml.

Please share your thoughts and comments. All of them are welcomed.

PS: 

Related background is here:
http://sourceware.org/ml/gdb-patches/2012-04/msg00971.html.

Some results from experiment:
If we use a string to store the plain xml file as below, the size of
the string is 1869 bytes.

static const char *raw_target_xml =
"<?xml version=\"1.0\"?>"
"<!-- Copyright (C) 2010, 2011 Free Software Foundation, Inc."
"     Copying and distribution of this file, with or without modification,"
"     are permitted in any medium without royalty provided the copyright"
"     notice and this notice are preserved.  -->"
"<!DOCTYPE target SYSTEM \"gdb-target.dtd\">"
"<target>"
"<feature name=\"org.gnu.gdb.arm.m-profile\">"
"  <reg name=\"r0\" bitsize=\"32\"/>"
"  <reg name=\"r1\" bitsize=\"32\"/>"
"  <reg name=\"r2\" bitsize=\"32\"/>"
"  <reg name=\"r3\" bitsize=\"32\"/>"
"  <reg name=\"r4\" bitsize=\"32\"/>"
"  <reg name=\"r5\" bitsize=\"32\"/>"
"  <reg name=\"r6\" bitsize=\"32\"/>"
"  <reg name=\"r7\" bitsize=\"32\"/>"
"  <reg name=\"r8\" bitsize=\"32\"/>"
"  <reg name=\"r9\" bitsize=\"32\"/>"
"  <reg name=\"r10\" bitsize=\"32\"/>"
"  <reg name=\"r11\" bitsize=\"32\"/>"
"  <reg name=\"r12\" bitsize=\"32\"/>"
"  <reg name=\"sp\" bitsize=\"32\" type=\"data_ptr\"/>"
"  <reg name=\"lr\" bitsize=\"32\"/>"
"  <reg name=\"pc\" bitsize=\"32\" type=\"code_ptr\"/>"
"  <reg name=\"xpsr\" bitsize=\"32\" regnum=\"25\"/>"
"</feature>"
"<feature name=\"org.gnu.gdb.arm.vfp\">"
"  <reg name=\"d0\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d1\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d2\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d3\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d4\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d5\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d6\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d7\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d8\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d9\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d10\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d11\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d12\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d13\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d14\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"d15\" bitsize=\"64\" type=\"ieee_double\"/>"
"  <reg name=\"fpscr\" bitsize=\"32\" type=\"int\" group=\"float\"/>"
"</feature>"
"</target>"
;

Then compress it with zlib function:

ret = compress2((Bytef*)dest, (uLongf *)&dlen, (const Bytef
*)raw_target_xml, (uLong)mlen, Z_BEST_COMPRESSION);

The size of compressed data is 462 bytes.

So Jonathan: is this size acceptable to eCos stub?

BR,
Terry

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Jonathan Larmour]

On 11/06/12 08:31, Terry Guo wrote:
> [snip] So a practical solution for such stub is to use the content
> of pre-compressed target.xml file and reply host gdb with that
> content. To cope with such compressed xml file, I am going to propose
> following working model:
> 1). Use ZLIB to compress (at Z_BEST_COMPRESSION level) and decompress
> the xml file in memory, don't use the gz format.

That's fine.

> 2). When reply to host gdb, the first four bytes of the packet should
> be "ZLIB" and following four bytes should be the length of data before
> compressed, the rest should be the compressed data. In this way, the
> host gdb can know the format of received data and how to decompress
> them.

We have to be concerned about compatibility here. I suspect current and
older GDBs may get awfully confused by a stub which just goes ahead and
returns compressed data instead of XML. It would be better to send nothing
and let GDB fall back on guesswork, than send compressed XML to a GDB
which can't support it.

It may want some variation on the qSupported "qXfer:features:read"
response instead. I'll defer to GDB maintainers in the choice, but perhaps
qXfer:features:zread ?

[snip]
> Some results from experiment:
> If we use a string to store the plain xml file as below, the size of
> the string is 1869 bytes.
[snip]
> The size of compressed data is 462 bytes.
> 
> So Jonathan: is this size acceptable to eCos stub?

Yes that's a big improvement. While I would still prefer the overhead to
be 0, this may have to be the best compromise.

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

Hi Jonathan,

[snip]
> 
> > 2). When reply to host gdb, the first four bytes of the packet should
> > be "ZLIB" and following four bytes should be the length of data
> before
> > compressed, the rest should be the compressed data. In this way, the
> > host gdb can know the format of received data and how to decompress
> > them.
> 
> We have to be concerned about compatibility here. I suspect current and
> older GDBs may get awfully confused by a stub which just goes ahead and
> returns compressed data instead of XML. It would be better to send
> nothing
> and let GDB fall back on guesswork, than send compressed XML to a GDB
> which can't support it.
> 
> It may want some variation on the qSupported "qXfer:features:read"
> response instead. I'll defer to GDB maintainers in the choice, but
> perhaps
> qXfer:features:zread ?
> 

Agree. The host gdb should inform stub that whether it can support
compressed xml. The qXfer:features:zread is ok to me.

> [snip]
> > Some results from experiment:
> > If we use a string to store the plain xml file as below, the size of
> > the string is 1869 bytes.
> [snip]
> > The size of compressed data is 462 bytes.
> >
> > So Jonathan: is this size acceptable to eCos stub?
> 
> Yes that's a big improvement. While I would still prefer the overhead
> to
> be 0, this may have to be the best compromise.
> 

But if you consider for future, it would be very possible that two different
cores share same g reply length. The xml file is still the most reliable and
natural way to inform exact target register info to host gdb. I believe you
also don't like to add more and more guesses in GDB.

BR,
Terry

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Yao Qi]

On 06/11/2012 10:42 PM, Jonathan Larmour wrote:
> It may want some variation on the qSupported "qXfer:features:read"
> response instead. I'll defer to GDB maintainers in the choice, but perhaps
> qXfer:features:zread ?

I'd like GDB keeps using qXfer:features:read:target.xml to fetch the xml
file, even it is compressed.  We need some changes on qSupported in both
sides,

  1.  GDB sends qSupported with new feature like
"$qSupported:multiprocess+;qRelocInsn;compressedXML+", means GDB
understands compressed xml,
  2.  Stub replies with ";compressedXML+" to GDB,
  3.  GDB sends "$qXfer:features:read:target.xml" as usual,
  4.  Stub replies with compressed xml.

If "compressedXML" doesn't appear in either #1 or #2, send target xml in
plain text as usual.

-- 
Yao (齐尧)

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Ulrich Weigand]

Yao Qi wrote:
> On 06/11/2012 10:42 PM, Jonathan Larmour wrote:
> > It may want some variation on the qSupported "qXfer:features:read"
> > response instead. I'll defer to GDB maintainers in the choice, but perhaps
> > qXfer:features:zread ?
> 
> I'd like GDB keeps using qXfer:features:read:target.xml to fetch the xml
> file, even it is compressed.  We need some changes on qSupported in both
> sides,

That still makes the result of qXfer:features:read:target.xml dependent
on global state ...   Why not simply support compressed files by having
the stub respond to
   qXfer:features:read:target.xml.gz
(etc) if it has a compressed file image?

Optionally we might still add a qSupported feature as you describe;
otherwise GDB might just probe whether the stub knows .gz files.

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Jonathan Larmour]

On 12/06/12 13:56, Ulrich Weigand wrote:
> Yao Qi wrote:
>> I'd like GDB keeps using qXfer:features:read:target.xml to fetch the xml
>> file, even it is compressed.  We need some changes on qSupported in both
>> sides,
> 
> That still makes the result of qXfer:features:read:target.xml dependent
> on global state ...   Why not simply support compressed files by having
> the stub respond to
>    qXfer:features:read:target.xml.gz
> (etc) if it has a compressed file image?
> 
> Optionally we might still add a qSupported feature as you describe;
> otherwise GDB might just probe whether the stub knows .gz files.

For what it's worth, that sounds appealing to me. Strictly Terry's
proposal wasn't a .true gz file but a gzipped stream. But that's easy to
avoid if we just choose to use the name convention target.xmlz or suchlike.

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

> -----Original Message-----
> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
> Sent: Wednesday, June 13, 2012 12:24 AM
> To: Ulrich Weigand
> Cc: Yao Qi; Terry Guo; gdb-patches@sourceware.org; tromey@redhat.com;
> Richard Earnshaw; 'Pedro Alves'; Joey Ye
> Subject: Re: [RFC] Enable GDB handle compressed target.xml returned by
> GDB stub
> 
> On 12/06/12 13:56, Ulrich Weigand wrote:
> > Yao Qi wrote:
> >> I'd like GDB keeps using qXfer:features:read:target.xml to fetch the
> xml
> >> file, even it is compressed.  We need some changes on qSupported in
> both
> >> sides,
> >
> > That still makes the result of qXfer:features:read:target.xml
> dependent
> > on global state ...   Why not simply support compressed files by
> having
> > the stub respond to
> >    qXfer:features:read:target.xml.gz
> > (etc) if it has a compressed file image?
> >
> > Optionally we might still add a qSupported feature as you describe;
> > otherwise GDB might just probe whether the stub knows .gz files.
> 
> For what it's worth, that sounds appealing to me. Strictly Terry's
> proposal wasn't a .true gz file but a gzipped stream. But that's easy
> to
> avoid if we just choose to use the name convention target.xmlz or
> suchlike.
> 
> Jifl

I am kind of lost on "gz file". Do you mean there is a real gz file and the
stub will do following things to response host gdb request?

1. stub open the real gz file and read it into buffer.
2. stub transmit the buffer to host gdb.
3. stub close the file

I looked into some open source gdb servers like openocd and stlink. I found
they just use a string to store the content of xml file, they don't have a
real xml file. I think this way consumes less flash space because it doesn't
need store big gz file header.

BR,
Terry

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

> -----Original Message-----
> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
> Sent: Wednesday, June 13, 2012 12:24 AM
> To: Ulrich Weigand
> Cc: Yao Qi; Terry Guo; gdb-patches@sourceware.org; tromey@redhat.com;
> Richard Earnshaw; 'Pedro Alves'; Joey Ye
> Subject: Re: [RFC] Enable GDB handle compressed target.xml returned by
> GDB stub
> 
> On 12/06/12 13:56, Ulrich Weigand wrote:
> > Yao Qi wrote:
> >> I'd like GDB keeps using qXfer:features:read:target.xml to fetch the
> xml
> >> file, even it is compressed.  We need some changes on qSupported in
> both
> >> sides,
> >
> > That still makes the result of qXfer:features:read:target.xml
> dependent
> > on global state ...   Why not simply support compressed files by
> having
> > the stub respond to
> >    qXfer:features:read:target.xml.gz
> > (etc) if it has a compressed file image?
> >
> > Optionally we might still add a qSupported feature as you describe;
> > otherwise GDB might just probe whether the stub knows .gz files.
> 
> For what it's worth, that sounds appealing to me. Strictly Terry's
> proposal wasn't a .true gz file but a gzipped stream. But that's easy
> to
> avoid if we just choose to use the name convention target.xmlz or
> suchlike.
> 
> Jifl

So the whole process can be simplified to:

Step 1: GDB just sends "qXfer:features:read:target.xml.gz" to stub to try to
get a compressed xml file.
Step 2: If stub supports compressed xml, then it sends it to GDB. Otherwise
nothing get replied.
Step 3: If GDB gets something, then it proceeds to decompress the xml file
and go ahead. If get nothing, GDB sends " qXfer:features:read:target.xml" to
try to request the plain xml file.

For GDB that doesn't support compressed xml file, the
"qXfer:features:read:target.xml.gz" will never be used.

How about the above model?

BR,
Terry

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Pedro Alves]

On 06/13/2012 12:57 PM, Terry Guo wrote:

> 
>> -----Original Message-----
>> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
>> Sent: Wednesday, June 13, 2012 12:24 AM
>> To: Ulrich Weigand
>> Cc: Yao Qi; Terry Guo; gdb-patches@sourceware.org; tromey@redhat.com;
>> Richard Earnshaw; 'Pedro Alves'; Joey Ye
>> Subject: Re: [RFC] Enable GDB handle compressed target.xml returned by
>> GDB stub
>>
>> On 12/06/12 13:56, Ulrich Weigand wrote:
>>> Yao Qi wrote:
>>>> I'd like GDB keeps using qXfer:features:read:target.xml to fetch the
>> xml
>>>> file, even it is compressed.  We need some changes on qSupported in
>> both
>>>> sides,
>>>
>>> That still makes the result of qXfer:features:read:target.xml
>> dependent
>>> on global state ...   Why not simply support compressed files by
>> having
>>> the stub respond to
>>>    qXfer:features:read:target.xml.gz
>>> (etc) if it has a compressed file image?
>>>
>>> Optionally we might still add a qSupported feature as you describe;
>>> otherwise GDB might just probe whether the stub knows .gz files.
>>
>> For what it's worth, that sounds appealing to me. Strictly Terry's
>> proposal wasn't a .true gz file but a gzipped stream. But that's easy
>> to
>> avoid if we just choose to use the name convention target.xmlz or
>> suchlike.
>>
>> Jifl
> 
> So the whole process can be simplified to:
> 
> Step 1: GDB just sends "qXfer:features:read:target.xml.gz" to stub to try to
> get a compressed xml file.
> Step 2: If stub supports compressed xml, then it sends it to GDB. Otherwise
> nothing get replied.
> Step 3: If GDB gets something, then it proceeds to decompress the xml file
> and go ahead. If get nothing, GDB sends " qXfer:features:read:target.xml" to
> try to request the plain xml file.
> 
> For GDB that doesn't support compressed xml file, the
> "qXfer:features:read:target.xml.gz" will never be used.
> 
> How about the above model?


Please don't consider only target.xml in the design, but also the xi:includes.

-- 
Pedro Alves

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

> >
> > So the whole process can be simplified to:
> >
> > Step 1: GDB just sends "qXfer:features:read:target.xml.gz" to stub to
> try to
> > get a compressed xml file.
> > Step 2: If stub supports compressed xml, then it sends it to GDB.
> Otherwise
> > nothing get replied.
> > Step 3: If GDB gets something, then it proceeds to decompress the xml
> file
> > and go ahead. If get nothing, GDB sends "
> qXfer:features:read:target.xml" to
> > try to request the plain xml file.
> >
> > For GDB that doesn't support compressed xml file, the
> > "qXfer:features:read:target.xml.gz" will never be used.
> >
> > How about the above model?
> 
> 
> Please don't consider only target.xml in the design, but also the
> xi:includes.
> 

Yes, we need to consider xi:includes which means we have to involve a global state. A refine to Yao's solution:

1. If host GDB support compressed xml file, then it sends "$qSupported:multiprocess+;qRelocInsn;compressedXML+" to stub. Here is a new feature compressedXML+.

2. If stub supports compressed xml file and will reply with compressed xml file, then it replies with ";compressedXML+" to GDB.

3. We still use "$qXfer:features:read:target.xml" to request xml file as usual, including the file in "xi:includes". For stub that returns ";compressedXML+", the gdb will assume all the returned xml file are compressed.

4. Otherwise the returned data are not compressed.

BR,
Terry

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Ulrich Weigand]

Terry Guo wrote:

> Yes, we need to consider xi:includes which means we have to involve a
> global state.

Not necessarily; the way I had intended my suggestion to work was that
GDB always adds ".gz" (or some other suffix if we actually are not
compatible with the .gz file format) to *every* file it fetches, not
just to the initial target.xml, but also to other files fetched via
xi:include statements ...

If the compressed version of the file is not available, GDB would
then fall back to the original file name (on a file-by-file basis).

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Jonathan Larmour]

On 13/06/12 02:57, Terry Guo wrote:
>> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
>>
>> For what it's worth, that sounds appealing to me. Strictly Terry's 
>> proposal wasn't a .true gz file but a gzipped stream. But that's
>> easy to avoid if we just choose to use the name convention
>> target.xmlz or suchlike.
> 
> I am kind of lost on "gz file". Do you mean there is a real gz file and the
> stub will do following things to response host gdb request?
> 
> 1. stub open the real gz file and read it into buffer.
> 2. stub transmit the buffer to host gdb.
> 3. stub close the file

No I thought in your initial message you were proposing a zlib compressed
stream, rather than a .gz file (which includes the gz file header/footer).
I think that proposal is better because it's true that the only things
really needed are the zlib compressed stream and the length. No .gz files
would be involved at all. So calling it target.xml.gz would be misleading
since it wouldn't contain the gzip header/footer required to make it a
true .gz format file.

> I looked into some open source gdb servers like openocd and stlink. I found
> they just use a string to store the content of xml file, they don't have a
> real xml file. I think this way consumes less flash space because it doesn't
> need store big gz file header.

Flash space isn't an issue for openOCD or ST-link. But yes, what I would
hope the end result woudl permit is that the target should only need to
hold a const char[] containing the compressed xml data, and the
decompressed size.

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Jonathan Larmour]

On 13/06/12 14:12, Ulrich Weigand wrote:
> Terry Guo wrote:
> 
>> Yes, we need to consider xi:includes which means we have to involve a
>> global state.
> 
> Not necessarily; the way I had intended my suggestion to work was that
> GDB always adds ".gz" (or some other suffix if we actually are not
> compatible with the .gz file format) to *every* file it fetches, not
> just to the initial target.xml, but also to other files fetched via
> xi:include statements ...
> 
> If the compressed version of the file is not available, GDB would
> then fall back to the original file name (on a file-by-file basis).

I agree.

This also means that the target can choose whether to return any file as
compressed or not, rather than having to have everything compressed from
then on, even if some XML files might take up more space compressed than
uncompressed, or might need to be generated - but not all files. It's
getting a bit hypothetical at this point, it's true, but not unreasonably so.

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

> -----Original Message-----
> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
> Sent: Wednesday, June 13, 2012 9:42 PM
> To: Terry Guo
> Cc: Yao Qi; gdb-patches@sourceware.org; tromey@redhat.com; Richard
> Earnshaw; 'Pedro Alves'; Joey Ye; Ulrich Weigand
> Subject: Re: [RFC] Enable GDB handle compressed target.xml returned by
> GDB stub
> 
> On 13/06/12 02:57, Terry Guo wrote:
> >> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
> >>
> >> For what it's worth, that sounds appealing to me. Strictly Terry's
> >> proposal wasn't a .true gz file but a gzipped stream. But that's
> >> easy to avoid if we just choose to use the name convention
> >> target.xmlz or suchlike.
> >
> > I am kind of lost on "gz file". Do you mean there is a real gz file
> and the
> > stub will do following things to response host gdb request?
> >
> > 1. stub open the real gz file and read it into buffer.
> > 2. stub transmit the buffer to host gdb.
> > 3. stub close the file
> 
> No I thought in your initial message you were proposing a zlib
> compressed
> stream, rather than a .gz file (which includes the gz file
> header/footer).
> I think that proposal is better because it's true that the only things
> really needed are the zlib compressed stream and the length. No .gz
> files
> would be involved at all. So calling it target.xml.gz would be
> misleading
> since it wouldn't contain the gzip header/footer required to make it a
> true .gz format file.
> 

Great. Then we are on the same page now.

BR,
Terry

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

> -----Original Message-----
> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
> Sent: Wednesday, June 13, 2012 9:47 PM
> To: Ulrich Weigand
> Cc: Terry Guo; 'Pedro Alves'; Yao Qi; gdb-patches@sourceware.org;
> tromey@redhat.com; Richard Earnshaw; Joey Ye
> Subject: Re: [RFC] Enable GDB handle compressed target.xml returned by
> GDB stub
> 
> On 13/06/12 14:12, Ulrich Weigand wrote:
> > Terry Guo wrote:
> >
> >> Yes, we need to consider xi:includes which means we have to involve
> a
> >> global state.
> >
> > Not necessarily; the way I had intended my suggestion to work was
> that
> > GDB always adds ".gz" (or some other suffix if we actually are not
> > compatible with the .gz file format) to *every* file it fetches, not
> > just to the initial target.xml, but also to other files fetched via
> > xi:include statements ...
> >
> > If the compressed version of the file is not available, GDB would
> > then fall back to the original file name (on a file-by-file basis).
> 
> I agree.
> 
> This also means that the target can choose whether to return any file
> as
> compressed or not, rather than having to have everything compressed
> from
> then on, even if some XML files might take up more space compressed
> than
> uncompressed, or might need to be generated - but not all files. It's
> getting a bit hypothetical at this point, it's true, but not
> unreasonably so.
> 

Thanks Ulrich and Jonathan, I think I get your points. Now the only
remaining question is about the suffix. How about just the .z, like
"qXfer:features:read:target.xml.z" which means the host gdb is trying to
request something compressed by zlib?

BR,
Terry

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Pedro Alves]

On 06/13/2012 02:12 PM, Ulrich Weigand wrote:

> Terry Guo wrote:
> 
>> Yes, we need to consider xi:includes which means we have to involve a
>> global state.
> 
> Not necessarily; the way I had intended my suggestion to work was that
> GDB always adds ".gz" (or some other suffix if we actually are not
> compatible with the .gz file format) to *every* file it fetches, not
> just to the initial target.xml, but also to other files fetched via
> xi:include statements ...
> 
> If the compressed version of the file is not available, GDB would
> then fall back to the original file name (on a file-by-file basis).


That sounds fine.  It makes gdb roundtrip to the target twice as
much for tdescs in the limit, but maybe that doesn't matter in
practice.

But note this scheme only works because we're fetching named xml files.
What of other xml objects that aren't filename based?  I guess it's
plausible that we'll find other situations where compressing xml
data would be useful.  In that perspective, something like the
'try qXfer:features:zread:target.xml first, then
qXfer:features:read:target.xml, etc.' alternative sounded attractive.
Or should we not bother?

-- 
Pedro Alves

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Joel Brobecker]

> In that perspective, something like the
> 'try qXfer:features:zread:target.xml first, then
> qXfer:features:read:target.xml, etc.' alternative sounded attractive.
> Or should we not bother?

Sounds more attractive indeed.

-- 
Joel

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Ulrich Weigand]

Joel Brobecker wrote:
> > In that perspective, something like the
> > 'try qXfer:features:zread:target.xml first, then
> > qXfer:features:read:target.xml, etc.' alternative sounded attractive.
> > Or should we not bother?
> 
> Sounds more attractive indeed.

The reason for suggesting just a different filename was simply
that this can use the existing infrastructure without much change.
On the other hand, :zread not only introduces a new packet type,
but in fact a completely new operation class for qXfer packets.

If this is going to be used just for the one special case, I'm
not sure it is worthwhile to bother with this.  If however we
feel that :zread could become useful for other situations
(e.g. other qXfer objects beside qXfer:features), then I certainly
don't object to adding it ...

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Jonathan Larmour]

On 14/06/12 11:14, Pedro Alves wrote:
> 
> That sounds fine.  It makes gdb roundtrip to the target twice as
> much for tdescs in the limit, but maybe that doesn't matter in
> practice.

We could still add the qSupported compressedXML+ aspect so that GDB won't
even try if the stub doesn't say it's going to do it.

> But note this scheme only works because we're fetching named xml files.
> What of other xml objects that aren't filename based? I guess it's
> plausible that we'll find other situations where compressing xml
> data would be useful.

Although it's meant to be a stream not a file as such - think of it more
like a protocol issue than a file transfer. And so far we've only been
talking about qXfer:features:read which is specific to target descriptions
(according to documented behaviour anyway). But hopefully this is
irrelevant.....

> In that perspective, something like the
> 'try qXfer:features:zread:target.xml first, then
> qXfer:features:read:target.xml, etc.' alternative sounded attractive.
> Or should we not bother?

I have a fresh variant proposal derived from the existing suggestions
which I think can accommodate all the concerns:

1) Remote stub can return compressedXML+ in qSupported response

2) The current syntax of qXfer:features:read is:
     qXfer:features:read:ANNEX:OFFSET,LENGTH
I suggest instead:
     qXfer:features:read:ANNEX:OFFSET,LENGTH[,Z]
where the ,Z is only attempted if the stub supports the compressedXML
remote protocol feature.

3) GDB will attempt to do the read with the ",Z". If the stub returns a
NUL character (the standard approach for unsupported packets in the remote
protocol), then GDB retries with just the normal request for plaintext:
     qXfer:features:read:ANNEX:OFFSET,LENGTH


This can in fact then be applied to all qXfer:*:read primitives if
desired, although I doubt it's useful for many at present. But since every
qXfer request goes through remote_read_qxfer() if anything it's easier to
support them all, than not to.

Anyone got a better proposal?

Presumably, by analogy, the equivalent should be put in place for
qXfer:*:write and remote_write_qxfer() I guess.

And just in response to a mail by Terry:
>  How about just the .z, like
> "qXfer:features:read:target.xml.z" which means the host gdb is trying to
> request something compressed by zlib?

If the above proposal isn't supported and instead we do use an extension
to do this, then I suggest avoiding .z as that is too similar to the .Z
standard unix compress format extension, especially when you consider
people being used to case-insensitive filenames on certain OS's.

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

> 
> > In that perspective, something like the
> > 'try qXfer:features:zread:target.xml first, then
> > qXfer:features:read:target.xml, etc.' alternative sounded attractive.
> > Or should we not bother?
> 
> I have a fresh variant proposal derived from the existing suggestions
> which I think can accommodate all the concerns:
> 
> 1) Remote stub can return compressedXML+ in qSupported response
> 
> 2) The current syntax of qXfer:features:read is:
>      qXfer:features:read:ANNEX:OFFSET,LENGTH
> I suggest instead:
>      qXfer:features:read:ANNEX:OFFSET,LENGTH[,Z]
> where the ,Z is only attempted if the stub supports the compressedXML
> remote protocol feature.
> 

I don't think it will work after I looked into some open source GDB servers like STLINk, OpenOCD. After they confirm that the packet is qXfer:features:read, they just extract information of ANNEX, OFFSET and LENGTH. Whether there is a "Z" doesn't impact their response behavior.

But the new "zread" can work because those existing gdb servers do check whether the op is "read". So for "zread", that check will fail and an error code will be returned to host gdb, thus host gdb knows the compressed xml file isn't supported. While for the new stub that knows "zread", the compressed xml file will be returned.

BR,
Terry

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Jonathan Larmour]

On 19/06/12 08:40, Terry Guo wrote:
> Jifl wrote:
>> 1) Remote stub can return compressedXML+ in qSupported response
>> 
>> 2) The current syntax of qXfer:features:read is: 
>> qXfer:features:read:ANNEX:OFFSET,LENGTH I suggest instead: 
>> qXfer:features:read:ANNEX:OFFSET,LENGTH[,Z] where the ,Z is only
>> attempted if the stub supports the compressedXML remote protocol
>> feature.
> 
> I don't think it will work after I looked into some open source GDB
> servers like STLINk, OpenOCD. After they confirm that the packet is
> qXfer:features:read, they just extract information of ANNEX, OFFSET and
> LENGTH. Whether there is a "Z" doesn't impact their response behavior.

Then they wouldn't have sent compressedXML+ in their qSupported response.
So they'll continue to work fine.

> But the new "zread" can work because those existing gdb servers do
> check whether the op is "read". So for "zread", that check will fail
> and an error code will be returned to host gdb, thus host gdb knows the
> compressed xml file isn't supported. While for the new stub that knows
> "zread", the compressed xml file will be returned.

Sure, but with my suggestion we save having to implement a new command on
the remote side, as well as avoiding a round trip of attempting the zread,
failing, and fallback to read every time.

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

> -----Original Message-----
> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
> Sent: Saturday, June 23, 2012 1:19 AM
> To: Terry Guo
> Cc: Pedro Alves; Ulrich Weigand; Yao Qi; gdb-patches@sourceware.org;
> tromey@redhat.com; Richard Earnshaw; Joey Ye
> Subject: Re: [RFC] Enable GDB handle compressed target.xml returned by
> GDB stub
> 
> On 19/06/12 08:40, Terry Guo wrote:
> > Jifl wrote:
> >> 1) Remote stub can return compressedXML+ in qSupported response
> >>
> >> 2) The current syntax of qXfer:features:read is:
> >> qXfer:features:read:ANNEX:OFFSET,LENGTH I suggest instead:
> >> qXfer:features:read:ANNEX:OFFSET,LENGTH[,Z] where the ,Z is only
> >> attempted if the stub supports the compressedXML remote protocol
> >> feature.
> >
> > I don't think it will work after I looked into some open source GDB
> > servers like STLINk, OpenOCD. After they confirm that the packet is
> > qXfer:features:read, they just extract information of ANNEX, OFFSET
> and
> > LENGTH. Whether there is a "Z" doesn't impact their response behavior.
> 
> Then they wouldn't have sent compressedXML+ in their qSupported
> response.
> So they'll continue to work fine.
> 
> > But the new "zread" can work because those existing gdb servers do
> > check whether the op is "read". So for "zread", that check will fail
> > and an error code will be returned to host gdb, thus host gdb knows
> the
> > compressed xml file isn't supported. While for the new stub that
> knows
> > "zread", the compressed xml file will be returned.
> 
> Sure, but with my suggestion we save having to implement a new command
> on
> the remote side, as well as avoiding a round trip of attempting the
> zread,
> failing, and fallback to read every time.
> 

Hi Jonathan,

Maybe I am not correctly understanding your proposal. Please let me repeat and feel free to correct me:

There are a bunch of "qXfer:xxxxx:read" inside GDB such as qXfer:features:read, qXfer:memory-map:read and so on. All of them are done by one function named remote_read_qxfer. I think it's better if all the "qXfer:xxxxx:read" can utilize the support of compressed xml file. It's possible that stub has compressed xml for "qXfer:features:read" but has plain xml for "qXfer:memory-map:read". So IMHO the round trip of attempting is unavoidable.

With zread solution, the new GDB will do following things:

   Send command "qXfer:xxxxx:zread" to stub
   If get NULL reply
   Then send normal command "qXfer:xxxxx:read" to stub

This solution can work with either existing stub or the new stub that support compressed xml file.

For your solution, I think the round trip attempting is still needed because stub might return compressed xml and plain xml file. Since we always need the round trip, then I think it might not be important that whether stub can return compressedXML+.

BR,
Terry

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

Hi Jonathan,

I read your email again and think your solution is:

  If stub doesn't return compressedXML+ then
    We go with plain read
  Else we go with compressed read
     If we get NUL reply
        Then we fall back to plain read

If this time my understanding on your solution is correct, I think it does save effort on round trip by not always round trip.

BR,
Terry 


> 
> > -----Original Message-----
> > From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
> > Sent: Saturday, June 23, 2012 1:19 AM
> > To: Terry Guo
> > Cc: Pedro Alves; Ulrich Weigand; Yao Qi; gdb-patches@sourceware.org;
> > tromey@redhat.com; Richard Earnshaw; Joey Ye
> > Subject: Re: [RFC] Enable GDB handle compressed target.xml returned
> by
> > GDB stub
> >
> > On 19/06/12 08:40, Terry Guo wrote:
> > > Jifl wrote:
> > >> 1) Remote stub can return compressedXML+ in qSupported response
> > >>
> > >> 2) The current syntax of qXfer:features:read is:
> > >> qXfer:features:read:ANNEX:OFFSET,LENGTH I suggest instead:
> > >> qXfer:features:read:ANNEX:OFFSET,LENGTH[,Z] where the ,Z is only
> > >> attempted if the stub supports the compressedXML remote protocol
> > >> feature.
> > >
> > > I don't think it will work after I looked into some open source GDB
> > > servers like STLINk, OpenOCD. After they confirm that the packet is
> > > qXfer:features:read, they just extract information of ANNEX, OFFSET
> > and
> > > LENGTH. Whether there is a "Z" doesn't impact their response
> behavior.
> >
> > Then they wouldn't have sent compressedXML+ in their qSupported
> > response.
> > So they'll continue to work fine.
> >
> > > But the new "zread" can work because those existing gdb servers do
> > > check whether the op is "read". So for "zread", that check will
> fail
> > > and an error code will be returned to host gdb, thus host gdb knows
> > the
> > > compressed xml file isn't supported. While for the new stub that
> > knows
> > > "zread", the compressed xml file will be returned.
> >
> > Sure, but with my suggestion we save having to implement a new
> command
> > on
> > the remote side, as well as avoiding a round trip of attempting the
> > zread,
> > failing, and fallback to read every time.
> >
> 
> Hi Jonathan,
> 
> Maybe I am not correctly understanding your proposal. Please let me
> repeat and feel free to correct me:
> 
> There are a bunch of "qXfer:xxxxx:read" inside GDB such as
> qXfer:features:read, qXfer:memory-map:read and so on. All of them are
> done by one function named remote_read_qxfer. I think it's better if
> all the "qXfer:xxxxx:read" can utilize the support of compressed xml
> file. It's possible that stub has compressed xml for
> "qXfer:features:read" but has plain xml for "qXfer:memory-map:read". So
> IMHO the round trip of attempting is unavoidable.
> 
> With zread solution, the new GDB will do following things:
> 
>    Send command "qXfer:xxxxx:zread" to stub
>    If get NULL reply
>    Then send normal command "qXfer:xxxxx:read" to stub
> 
> This solution can work with either existing stub or the new stub that
> support compressed xml file.
> 
> For your solution, I think the round trip attempting is still needed
> because stub might return compressed xml and plain xml file. Since we
> always need the round trip, then I think it might not be important that
> whether stub can return compressedXML+.
> 
> BR,
> Terry

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Jonathan Larmour]

Terry Guo wrote:
> Hi Jonathan,
> 
> I read your email again and think your solution is:
> 
>   If stub doesn't return compressedXML+ then
>     We go with plain read
>   Else we go with compressed read
>      If we get NUL reply
>         Then we fall back to plain read
> 
> If this time my understanding on your solution is correct, I think it
> does save effort on round trip by not always round trip.

Yes, that's what I was proposing.

By the way, there are a lot of people on CC for this thread, and I don't 
know if they ought to be. Send me and Terry a direct note if you want off 
CC - I've just been preserving the CCs.

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine

RE: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Terry Guo]

Hi Jonathan,

In terms of transferring the compressed xml file from stub to gdb, I find there are two methods:

1). Use the original binary format and escape special characters like "$", "#" and "}".
2). Convert the original binary format into ASCII format, such as the binary format(in hex) "01020304" will be converted into string of bytes "01020304"(in ASCII format). The size of binary format is 32 bits while the size of string format will be 64 bits (without the NULL terminator). The problem is that the binary format is much smaller than the original xml file, but after convert, we need to use bigger runtime buffer to store the string of bytes.

So which method will you prefer? I think the method 1) is better.

BR,
Terry

> -----Original Message-----
> From: Jonathan Larmour [mailto:jifl@ecoscentric.com]
> Sent: Tuesday, June 26, 2012 7:58 PM
> To: Terry Guo
> Cc: 'Pedro Alves'; 'Ulrich Weigand'; 'Yao Qi'; gdb-
> patches@sourceware.org; tromey@redhat.com; Richard Earnshaw; Joey Ye;
> Matthew Gretton-Dann
> Subject: Re: [RFC] Enable GDB handle compressed target.xml returned by
> GDB stub
> 
> Terry Guo wrote:
> > Hi Jonathan,
> >
> > I read your email again and think your solution is:
> >
> >   If stub doesn't return compressedXML+ then
> >     We go with plain read
> >   Else we go with compressed read
> >      If we get NUL reply
> >         Then we fall back to plain read
> >
> > If this time my understanding on your solution is correct, I think it
> > does save effort on round trip by not always round trip.
> 
> Yes, that's what I was proposing.
> 
> By the way, there are a lot of people on CC for this thread, and I
> don't
> know if they ought to be. Send me and Terry a direct note if you want
> off
> CC - I've just been preserving the CCs.
> 
> Jifl
> --
> eCosCentric Limited      http://www.eCosCentric.com/     The eCos
> experts
> Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223
> 245571
> Registered in England and Wales: Reg No 4422071.
> ------["Si fractum non sit, noli id reficere"]------
> Opinions==mine

Re: [RFC] Enable GDB handle compressed target.xml returned by GDB stub

[Jonathan Larmour]

On 27/06/12 04:42, Terry Guo wrote:
> Hi Jonathan,
> 
> In terms of transferring the compressed xml file from stub to gdb, I
> find there are two methods:
> 
> 1). Use the original binary format and escape special characters like "$", "#" and "}".
> 2). Convert the original binary format into ASCII format, such as the
> binary format(in hex) "01020304" will be converted into string of bytes
> "01020304"(in ASCII format). The size of binary format is 32 bits while
> the size of string format will be 64 bits (without the NULL terminator).
> The problem is that the binary format is much smaller than the original
> xml file, but after convert, we need to use bigger runtime buffer to store
> the string of bytes.

> So which method will you prefer? I think the method 1) is better.

remote_read_qxfer()/remote_write_qxfer() already do (1) so I don't see any
reason we'd want to allow (2).

Jifl
-- 
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
------["Si fractum non sit, noli id reficere"]------       Opinions==mine