[RFA/testsuite] Workaround timeout in default.exp

[RFA/testsuite] Workaround timeout in default.exp

[Joel Brobecker]

This is a problem that has been bugging me on AIX for as long as I can
remember: default.exp on AIX was taking hours to complete. So far, I've
just worked-around the problem by deactivating this test on that
platform, always postponing the task of looking at the source of the
problem for later.

Turns out we have two commands that have the gdb prompt in their
output, and this is somehow confusing test_gdb and test_gdb_multiple
into a timeout. I couldn't understand exactly why this was happening
for lack of time. I'm wondering if this might be an expect or dejagnu
bug, as it works fine on x86-linux for instance...

The attached patch works-around the problem by using the send_gdb and
gdb_expect pair instead of test_gdb. This short-circuits the smart
machinery behind test_gdb, and allows runtest to match the output
from the 2 commands correctly. With this patch, the testscase completes
successfully within a reasonable amount of time.

I added some comments explaining what was happening with test_gdb and
why we use send_gdb and gdb_expect instead, to make it clear why I made
this change.

2004-05-18  J. Brobecker  <brobecker@gnat.com>

	* gdb.base/default.exp: Rewrite a couple of tests to work-around
	a problem that causes this test and all the following tests to
	timeout.

Tested on AIX 5.1 and x86-linux. Would that be OK to apply?

-- 
Joel
Attachment:
default.exp.diff
Description: Text document

Re: [RFA/testsuite] Workaround timeout in default.exp

[Daniel Jacobowitz]

On Tue, May 18, 2004 at 02:01:47PM -0700, Joel Brobecker wrote:
> This is a problem that has been bugging me on AIX for as long as I can
> remember: default.exp on AIX was taking hours to complete. So far, I've
> just worked-around the problem by deactivating this test on that
> platform, always postponing the task of looking at the source of the
> problem for later.
> 
> Turns out we have two commands that have the gdb prompt in their
> output, and this is somehow confusing test_gdb and test_gdb_multiple
> into a timeout. I couldn't understand exactly why this was happening
> for lack of time. I'm wondering if this might be an expect or dejagnu
> bug, as it works fine on x86-linux for instance...
> 
> The attached patch works-around the problem by using the send_gdb and
> gdb_expect pair instead of test_gdb. This short-circuits the smart
> machinery behind test_gdb, and allows runtest to match the output
> from the 2 commands correctly. With this patch, the testscase completes
> successfully within a reasonable amount of time.
> 
> I added some comments explaining what was happening with test_gdb and
> why we use send_gdb and gdb_expect instead, to make it clear why I made
> this change.
> 
> 2004-05-18  J. Brobecker  <brobecker@gnat.com>
> 
> 	* gdb.base/default.exp: Rewrite a couple of tests to work-around
> 	a problem that causes this test and all the following tests to
> 	timeout.
> 
> Tested on AIX 5.1 and x86-linux. Would that be OK to apply?

Rather than adding a FIXME, let me try to explain what is going on.

The first part is a guess.  For some reason, the pattern "\(gdb\) $"
never matches on GNU/Linux but does on AIX and Solaris.  I presume this
is because of some difference in the pseudo-TTY layer or in the
standard I/O library, which causes characters to be send in smaller
batches.  So at some point on those systems, the buffer ends with the
prompt.

So the second part is how to prevent the prompt pattern from matching. 
The way to do it is use gdb_test_multiple, and add an explicit
condition for the text which comes _before_ the extra prompt.

I'm doing this from memory, so it will need some adjustment, but you
can do it sort of like this.  The "global" lines may not be necessary,
so try without them.

global saw_good_text
set saw_good_text 0

gdb_test_multiple "command" "test name" {
  -re "long\ngood\ntext\nwithout\ntrailing dot, star, or prompt" {
    global saw_good_text
    set saw_good_text 1
    exp_continue
  }
  -re "prompt is \"$gdb_prompt" {
    exp_continue
  }
  -re "$gdb_prompt $" {
    global saw_good_text
    if {$saw_good_text} {
      pass "test name"
    } else {
      fail "test name (unknown output)"
    }
  }
}

This makes sure to consume both the bad prompt and the good prompt.

-- 
Daniel Jacobowitz

Re: [RFA/testsuite] Workaround timeout in default.exp

[Joel Brobecker]

> > 2004-05-18  J. Brobecker  <brobecker@gnat.com>
> > 
> > 	* gdb.base/default.exp: Rewrite a couple of tests to work-around
> > 	a problem that causes this test and all the following tests to
> > 	timeout.
> > 
> > Tested on AIX 5.1 and x86-linux. Would that be OK to apply?
> 
> Rather than adding a FIXME, let me try to explain what is going on.

OK.

> The first part is a guess.  For some reason, the pattern "\(gdb\) $"
> never matches on GNU/Linux but does on AIX and Solaris.  I presume this
> is because of some difference in the pseudo-TTY layer or in the
> standard I/O library, which causes characters to be send in smaller
> batches.  So at some point on those systems, the buffer ends with the
> prompt.

There may be a crack in your theory, because there something it doesn't
explain. When I use the simple send_gdb/gdb_expect sequence, then I get
the whole output, ie the differences you are mentioning do not come into
play. I verified this by using expect_out.

I tried your suggestion, without much luck so far. Here is what I have
tried:

    print "DEBUG: Starting test...."
    gdb_test_multiple "info set" "info set" {
      -re "confirm:  Whether to confirm potentially dangerous operations is o\[a-z\]*.(\[^\r\n\]*\[\r\n\])+history filename:  The filename in which to record the command history is (\[^\r\n\]*\[\r\n\])+listsize:  Number of source lines gdb will list by default is 10.*prompt:  Gdb's " {
        global saw_good_text
        set saw_good_text 1
        print "DEBUG: First part of output found."
        exp_continue
      }
      -re "prompt is \"$gdb_prompt \"" {
        print "DEBUG: Line with GDB prompt."
        exp_continue
      }
      -re "$gdb_prompt $" {
        global saw_good_text
        print "DEBUG: Final part of output."
        if {$saw_good_text} {
          pass "test name"
        } else {
          fail "test name (unknown output)"
        }
      }
    }

But none of the regexp matched (no DEBUG: output besides the first
one before gdb_test_multiple), and it eventually timedout.

I tried with verbose set to 3, and that gives me the following output.
First, it prints:

        Running ./gdb.base/joel.exp ...
        DEBUG: Starting test....
        Sending "info set" to gdb
        Looking to match ""confirm:  Whether to confirm potentially dangerous operations is o[a-z]*.([^\r\n]*[\r\n])+history filename:  The filename in which to record the command history is ([^\r\n]*[\r\n])+listsize:  Number of source lines gdb will list by default is 10.*prompt:  Gdb's "; "prompt is "\(gdb\) ""; "\(gdb\) $""
        Message is "info set"
        board_info host exists name
        board_info host name 
        getting cam name
        board_info cam exists name
        board_info cam exists protocol
        board_info cam generic_name 
        getting cam generic_name
        call_remote  send cam {info set
        } 
        board_info cam file_transfer 
        getting cam file_transfer
        board_info cam connect 
        getting cam connect
        call_remote calling standard_send
        board_info cam exists fileid
        board_info cam fileid 
        getting cam fileid
        shell_id in standard_send is 6
        board_info cam fileid 
        getting cam fileid
        send -i 6 -- {info set
        }
        board_info cam fileid 
        getting cam fileid
        board_info target exists gdb,timeout
        board_info host fileid 
        getting cam fileid

At which point we get stuck for a while, and then we receive the rest
of the output:

        FAIL: gdb.base/joel.exp: info set (timeout)
        Quitting /cam.a/brobecke/gdb-public/gdb/testsuite/../../gdb/gdb -nx
        [rest of output snip'ed]

-- 
Joel

Re: [RFA/testsuite] Workaround timeout in default.exp

[Daniel Jacobowitz]

On Wed, May 19, 2004 at 06:48:11PM -0700, Joel Brobecker wrote:
> > > 2004-05-18  J. Brobecker  <brobecker@gnat.com>
> > > 
> > > 	* gdb.base/default.exp: Rewrite a couple of tests to work-around
> > > 	a problem that causes this test and all the following tests to
> > > 	timeout.
> > > 
> > > Tested on AIX 5.1 and x86-linux. Would that be OK to apply?
> > 
> > Rather than adding a FIXME, let me try to explain what is going on.
> 
> OK.
> 
> > The first part is a guess.  For some reason, the pattern "\(gdb\) $"
> > never matches on GNU/Linux but does on AIX and Solaris.  I presume this
> > is because of some difference in the pseudo-TTY layer or in the
> > standard I/O library, which causes characters to be send in smaller
> > batches.  So at some point on those systems, the buffer ends with the
> > prompt.
> 
> There may be a crack in your theory, because there something it doesn't
> explain. When I use the simple send_gdb/gdb_expect sequence, then I get
> the whole output, ie the differences you are mentioning do not come into
> play. I verified this by using expect_out.
> 
> I tried your suggestion, without much luck so far. Here is what I have
> tried:

If you're trying to debug expect matching, I recommend "exp_internal 1"
before the block (and maybe "exp_internal 0" after).  That will let you
see exactly what's going on, and why my crackpot theory is wrong :)

-- 
Daniel Jacobowitz

Re: [RFA/testsuite] Workaround timeout in default.exp

[Joel Brobecker]

> If you're trying to debug expect matching, I recommend "exp_internal 1"
> before the block (and maybe "exp_internal 0" after).  That will let you
> see exactly what's going on, and why my crackpot theory is wrong :)

Nice command, that really helped understand better what was going on.

Roughly, it seems that what is hurting us is that expect is eating up
the output from GDB reeeaaaal slooooowwww. Not expect's fault, I imagine,
but by counting characters in the first few iterations, it looks like
expect gets the output 12-15 characters at a time. It takes a total
of 113 reads to fetch the entire command output, which contains ~9000
characters, which makes around 80 chars per read.

And from watching it run and spit the expect debug data, it seems as if
expect is getting slower and slower at each iteration. It looks like an
algorithm that's not linear, but something exponential instead.

As an experiment, I tried increasing the timeout value to 600secs,
and the test did complete successfully... After 18mins!

It seemed to be slowing down on the evaluation of one specific regexp:

        ".*\(gdb\) $"? no

So I uncommented it in gdb_test_multiple as follow, just to see what
effect it would have:

        #  -re ".*$gdb_prompt $" {
        #     if ![string match "" $message] then {
        #       fail "$message"
        #     }
        #     set result 1
        # }

You are going to be surprised. That specific test takes less than 40secs
to run. The whole default.exp test takes a total of 1mn45secs.

And what's curious, even after have is that I still see expect try the
same regexp. Before commenting out the code above, the sequence of
regexp evaluations looked like:

        "Undefined[a-z]* command:.*\(gdb\) $"? no
        "Ambiguous command.*\(gdb\) $"? no
        "Program exited with code [0-9]+.*\(gdb\) $"? no
        "EXIT code [0-9\r\n]+Program exited normally.*\(gdb\) $"? no
        "The program is not being run.*\(gdb\) $"? no
        ".*\(gdb\) $"? no
        "<return>"? no
        "\(y or n\) "? no

After, it was slightly transformed into:

        "Undefined[a-z]* command:.*\(gdb\) $"? no
        "Ambiguous command.*\(gdb\) $"? no
        "Program exited with code [0-9]+.*\(gdb\) $"? no
        "EXIT code [0-9\r\n]+Program exited normally.*\(gdb\) $"? no
        "The program is not being run.*\(gdb\) $"? no
        "#"? no
        ".*\(gdb\) $"? no
        "<return>"? no
        "\(y or n\) "? no

At this point, no comprendo anymore... :-/
(where does the "#" come from, and why is the gdb_prompt check
still there?)

-- 
Joel

Re: [RFA/testsuite] Workaround timeout in default.exp

[Daniel Jacobowitz]

On Thu, May 20, 2004 at 06:14:43PM -0700, Joel Brobecker wrote:
> > If you're trying to debug expect matching, I recommend "exp_internal 1"
> > before the block (and maybe "exp_internal 0" after).  That will let you
> > see exactly what's going on, and why my crackpot theory is wrong :)
> 
> Nice command, that really helped understand better what was going on.
> 
> Roughly, it seems that what is hurting us is that expect is eating up
> the output from GDB reeeaaaal slooooowwww. Not expect's fault, I imagine,
> but by counting characters in the first few iterations, it looks like
> expect gets the output 12-15 characters at a time. It takes a total
> of 113 reads to fetch the entire command output, which contains ~9000
> characters, which makes around 80 chars per read.
> 
> And from watching it run and spit the expect debug data, it seems as if
> expect is getting slower and slower at each iteration. It looks like an
> algorithm that's not linear, but something exponential instead.
> 
> As an experiment, I tried increasing the timeout value to 600secs,
> and the test did complete successfully... After 18mins!
> 
> It seemed to be slowing down on the evaluation of one specific regexp:
> 
>         ".*\(gdb\) $"? no

So, it sounds like that's the problem.  I don't know why expect would
behave this terribly, but I do know something else: the regular
expression is not front-anchored.  So the leading .* does nothing.  You
may as well try removing it.  I bet that will also fix the speed
problem.

[I think there should be a \r\n in front of the "$gdb_prompt $" anyway,
don't you?  Wouldn't that fix the related problem?  Please try
replacing that pattern with "\r\n$gdb_prompt $".]

> So I uncommented it in gdb_test_multiple as follow, just to see what
> effect it would have:
> 
>         #  -re ".*$gdb_prompt $" {
>         #     if ![string match "" $message] then {
>         #       fail "$message"
>         #     }
>         #     set result 1
>         # }
> 
> You are going to be surprised. That specific test takes less than 40secs
> to run. The whole default.exp test takes a total of 1mn45secs.
> 
> And what's curious, even after have is that I still see expect try the
> same regexp. Before commenting out the code above, the sequence of
> regexp evaluations looked like:
> 
>         "Undefined[a-z]* command:.*\(gdb\) $"? no
>         "Ambiguous command.*\(gdb\) $"? no
>         "Program exited with code [0-9]+.*\(gdb\) $"? no
>         "EXIT code [0-9\r\n]+Program exited normally.*\(gdb\) $"? no
>         "The program is not being run.*\(gdb\) $"? no
>         ".*\(gdb\) $"? no
>         "<return>"? no
>         "\(y or n\) "? no
> 
> After, it was slightly transformed into:
> 
>         "Undefined[a-z]* command:.*\(gdb\) $"? no
>         "Ambiguous command.*\(gdb\) $"? no
>         "Program exited with code [0-9]+.*\(gdb\) $"? no
>         "EXIT code [0-9\r\n]+Program exited normally.*\(gdb\) $"? no
>         "The program is not being run.*\(gdb\) $"? no
>         "#"? no
>         ".*\(gdb\) $"? no
>         "<return>"? no
>         "\(y or n\) "? no
> 
> At this point, no comprendo anymore... :-/
> (where does the "#" come from, and why is the gdb_prompt check
> still there?)

'#' doesn't do what you think it does.

First of all, it is not an element of TCL syntax.  It's just a command
which does nothing.  Secondly, as a consequence, you can't use it where
you aren't expecting commands.  So what you did was match "#" {action
body is "-re"}, and then ".*\(gdb\) $" {note that there's no leading
-re now, so it is treated as an exact match instead of a regular
expression!}.

-- 
Daniel Jacobowitz

Re: [RFA/testsuite] Workaround timeout in default.exp

[Joel Brobecker]

> > It seemed to be slowing down on the evaluation of one specific regexp:
> > 
> >         ".*\(gdb\) $"? no
> 
> So, it sounds like that's the problem.  I don't know why expect would
> behave this terribly, but I do know something else: the regular
> expression is not front-anchored.  So the leading .* does nothing.  You
> may as well try removing it.  I bet that will also fix the speed
> problem.
> 
> [I think there should be a \r\n in front of the "$gdb_prompt $" anyway,
> don't you?  Wouldn't that fix the related problem?  Please try
> replacing that pattern with "\r\n$gdb_prompt $".]

You are going to be happy: With "\r\n$gdb_prompt $" it only takes 30secs
to run the entire default.exp testcases. Down from 18mins in the best
of case, several hours worth of timeouts with nothing changed. :-).

I am about to test this change against the entire testsuite, to make
sure all is well...

> '#' doesn't do what you think it does.
> 
> First of all, it is not an element of TCL syntax.  It's just a command
> which does nothing.  Secondly, as a consequence, you can't use it where
> you aren't expecting commands.  So what you did was match "#" {action
> body is "-re"},

Argh! I think that's the second time I get caught by this. And since
my editor (vim) colored it in the usual comment color, I dind't pay
attention to this. 

> and then ".*\(gdb\) $" {note that there's no leading
> -re now, so it is treated as an exact match instead of a regular
> expression!}.

Which, if I understand correctly, tells me that the speedup I observed
was accidental :-).

-- 
Joel

Re: [RFA/testsuite] Workaround timeout in default.exp

[Jim Blandy]

Darn.  This discussion sounded familiar; it turns out I had a patch
for this written back in 2002 that I guess I never submitted.  Here it
is, adapted for the current sources; default.exp runs on AIX with this
patch.

(Your solution seems simpler.)

Sun Feb  3 19:52:31 2002  Jim Blandy  <jimb@seadog.cygnus.com>

	* gdb.base/default.exp: Distinguish the GDB prompt as it appears
	in "info set" and "show" output from the true prompt.

Index: gdb/testsuite/gdb.base/default.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.base/default.exp,v
retrieving revision 1.17
diff -c -p -r1.17 default.exp
*** gdb/testsuite/gdb.base/default.exp	20 Mar 2003 14:45:50 -0000	1.17
--- gdb/testsuite/gdb.base/default.exp	21 May 2004 06:48:57 -0000
*************** gdb_test "info s" "No stack." "info stac
*** 335,343 ****
  gdb_test "info stack" "No stack." "info stack"
  #test info set
  # FIXME -- needs to match the entire output
! # FIXME -- on native solaris 2.8, this test fails due to this line:
! #  prompt:  Gdb's prompt is "(gdb) ".^M
! gdb_test "info set" "confirm:  Whether to confirm potentially dangerous operations is o\[a-z\]*.(\[^\r\n\]*\[\r\n\])+history filename:  The filename in which to record the command history is (\[^\r\n\]*\[\r\n\])+listsize:  Number of source lines gdb will list by default is 10.*" "info set"
  gdb_test "info symbol" "Argument required .address.."
  #test info source
  gdb_test  "info source" "No current source file..*" "info source"
--- 335,373 ----
  gdb_test "info stack" "No stack." "info stack"
  #test info set
  # FIXME -- needs to match the entire output
! send_gdb "info set\n"
! set seen_confirm 0
! set seen_history 0
! set seen_prompt 0
! set seen_listsize 0
! gdb_expect {
!     -re ".*confirm:  Whether to confirm potentially dangerous operations is" {
! 	set seen_confirm 1
! 	exp_continue
!     }
!     -re ".*history filename:  The filename in which to record the command history is" {
! 	set seen_history 1
! 	exp_continue
!     }
!     -re ".*listsize:  Number of source lines gdb will list by default is 10" {
! 	set seen_listsize 1
! 	exp_continue
!     }
!     -re ".*prompt:  Gdb's prompt is \"${gdb_prompt}" {
! 	set seen_prompt 1
! 	exp_continue
!     }
!     -re ".*\[\r\n\]+${gdb_prompt} $" {
!     }
!     timeout {
! 	fail "info set (timeout)"
!     }
! }
! if {$seen_confirm && $seen_history && $seen_prompt && $seen_listsize} {
!     pass "info set"
! } else {
!     fail "info set"
! }
  gdb_test "info symbol" "Argument required .address.."
  #test info source
  gdb_test  "info source" "No current source file..*" "info source"
*************** gdb_test "show width" "Number of charact
*** 674,680 ****
  # This is only supported on targets which use exec.o.
  gdb_test "show write" "Writing into executable and core files is o.*" "show write"
  #test show
! gdb_test "show" "confirm:  *Whether to confirm potentially dangerous operations is on.(\[^\r\n\]*\[\r\n\])+history filename:  *The filename in which to record the command history is (\[^\r\n\]*\[\r\n\])+history save:  *Saving of the history record on exit is on.(\[^\r\n\]*\[\r\n\])+history size:  *The size of the command history is(\[^\r\n\]*\[\r\n\])+listsize:  *Number of source lines gdb will list by default is 10(\[^\r\n]*\[\r\n\])+print elements:  *Limit on string chars or array elements to print is 200..*" "show"
  #test stepi "si" abbreviation
  gdb_test  "si" "The program is not being run." "stepi \"si\" abbreviation"
  #test stepi
--- 704,758 ----
  # This is only supported on targets which use exec.o.
  gdb_test "show write" "Writing into executable and core files is o.*" "show write"
  #test show
! send_gdb "show\n"
! set seen_confirm 0
! set seen_history_filename 0
! set seen_history_save 0
! set seen_history_size 0
! set seen_listsize 0
! set seen_print_elements 0
! set seen_prompt 0
! gdb_expect {
!     -re ".*confirm:  *Whether to confirm potentially dangerous operations is" {
! 	set seen_confirm 1
! 	exp_continue
!     }
!     -re ".*history filename:  *The filename in which to record the command" {
! 	set seen_history_filename 1
! 	exp_continue
!     }
!     -re ".*history save:  *Saving of the history record on exit is" {
! 	set seen_history_save 1
! 	exp_continue
!     }
!     -re ".*history size:  *The size of the command history is" {
! 	set seen_history_size 1
! 	exp_continue
!     }
!     -re ".*listsize:  *Number of source lines gdb will list by default is" {
! 	set seen_listsize 1
! 	exp_continue
!     }
!     -re ".*print elements:  *Limit on string chars or array elements to" {
! 	set seen_print_elements 1
! 	exp_continue
!     }
!     -re "prompt:  Gdb's prompt is \"${gdb_prompt}" {
! 	set seen_prompt 1
! 	exp_continue
!     }
!     -re ".*\[\r\n\]+${gdb_prompt} $" {
!     }
!     timeout {
! 	fail "show (timeout)"
!     }
! }
! if {$seen_confirm && $seen_history_filename && $seen_history_save && $seen_history_size && $seen_listsize && $seen_print_elements && $seen_prompt} {
!     pass "show"
! } else {
!     fail "show"
! }
! 
  #test stepi "si" abbreviation
  gdb_test  "si" "The program is not being run." "stepi \"si\" abbreviation"
  #test stepi

Re: [RFA/testsuite] Workaround timeout in default.exp

[Joel Brobecker]

> Darn.  This discussion sounded familiar; it turns out I had a patch
> for this written back in 2002 that I guess I never submitted.  Here it
> is, adapted for the current sources; default.exp runs on AIX with this
> patch.
> 
> (Your solution seems simpler.)

The comment was slightly misleading. In your patch, you did essentially
the same as what I did (avoid using gdb_test), except that I didn't
break the expected output into several chunks as you did. I am guessing
it was to be able to isolate the line containing the GDB prompt, but
maybe that was because you wanted this test to be powerful enough to
match the lines the test is looking for in any order?

In any case, I think Daniel found the real problem.

> Sun Feb  3 19:52:31 2002  Jim Blandy  <jimb@seadog.cygnus.com>
> 
> 	* gdb.base/default.exp: Distinguish the GDB prompt as it appears
> 	in "info set" and "show" output from the true prompt.

-- 
Joel

Re: [RFA/testsuite] Workaround timeout in default.exp

[Daniel Jacobowitz]

On Fri, May 21, 2004 at 09:05:54AM -0700, Joel Brobecker wrote:
> > [I think there should be a \r\n in front of the "$gdb_prompt $" anyway,
> > don't you?  Wouldn't that fix the related problem?  Please try
> > replacing that pattern with "\r\n$gdb_prompt $".]
> 
> This works very well indeed.
> 
> 2004-05-21  Joel Brobecker  <brobecker@gnat.com>
>             Daniel Jacobowitz  <drow@mvista.com>
> 
>         * lib/gdb.exp (gdb_test_multiple): Improve regexp matching the
>         GDB prompt.
> 
> Tested on alpha-osf 5.1, and ppc-aix 5.1, and x86-linux.
> OK to apply?

Yes.  Thanks for testing it.

-- 
Daniel Jacobowitz

Re: [RFA/testsuite] Workaround timeout in default.exp

[Joel Brobecker]

> > 2004-05-21  Joel Brobecker  <brobecker@gnat.com>
> >             Daniel Jacobowitz  <drow@mvista.com>
> > 
> >         * lib/gdb.exp (gdb_test_multiple): Improve regexp matching the
> >         GDB prompt.
> > 
> > Tested on alpha-osf 5.1, and ppc-aix 5.1, and x86-linux.
> > OK to apply?
> 
> Yes.  Thanks for testing it.

Thanks. Checked in.

-- 
Joel