[REPOST][PATCH v4] H8/300: sim: Fix simulator hang caused by qsort on Windows/MinGW

[REPOST][PATCH v4] H8/300: sim: Fix simulator hang caused by qsort on Windows/MinGW

[Jan Dubiec]

On Windows, qsort() uses an unstable sorting algorithm, which results
in a "shuffled" opcode table rather than a properly sorted one, causing
the entire simulator to hang.

The simulator happens to work on Linux, but this behavior is not
guaranteed, because the glibc documentation clearly states that "If
two elements compare equal, their order after sorting is unpredictable."

This patch introduces two additional sort keys to the instruction
comparator function, making the resulting opcode table as close as
possible to the one that would be produced by a stable sorting algorithm.

It also fixes an issue in the instruction decoder where, for example,
mov.l @er7+,er1 is recognized as ldm.l @er7+,(er0-er1) if the opcode
table is sorted "the wrong way", i.e. when ldm appears before mov
in the table.

No regressions were found.

---
 sim/h8300/compile.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 49 insertions(+), 2 deletions(-)

diff --git a/sim/h8300/compile.c b/sim/h8300/compile.c
index 06988095228..0feb58fe183 100644
--- a/sim/h8300/compile.c
+++ b/sim/h8300/compile.c
@@ -509,7 +509,15 @@ decode (SIM_DESC sd, sim_cpu *cpu, int addr, unsigned char *data, decoded_inst *
 		}
 	      else if ((looking_for & MODE) == DATA)
 		{
-		  ;			/* Skip embedded data.  */
+		  /* Special case for ldm.l and stm.l. Otherwise, if the
+		    h8_opcodes[] table is sorted in a particular order,
+		    the instruction decoder may incorrectly recognize
+		    e.g. mov.l @er7+,er1 as ldm.l @er7+,(er0-er1).  */
+		  if (((OP_KIND (q->how) == O_LDM) ||
+		       (OP_KIND (q->how) == O_STM)) &&
+		      ((thisnib < 1) || (thisnib > 3)))
+		    goto fail;
+		  /* Skip embedded data for the rest.  */
 		}
 	      else if ((looking_for & MODE) == DBIT)
 		{
@@ -1586,6 +1594,7 @@ instruction_comparator (const void *p1_, const void *p2_)
 {
   struct h8_opcode *p1 = (struct h8_opcode *)p1_;
   struct h8_opcode *p2 = (struct h8_opcode *)p2_;
+  int cmp;
 
   /* The 1st sort key is based on whether or not the
      instruction is even available.  This reduces the
@@ -1605,7 +1614,45 @@ instruction_comparator (const void *p1_, const void *p2_)
     return p2_available - p1_available;
 
   /* Secondarily sort based on the first opcode nibble.  */
-  return p1->data.nib[0] - p2->data.nib[0];
+  if (p1->data.nib[0] != p2->data.nib[0])
+    return p1->data.nib[0] - p2->data.nib[0];
+
+  /* The 3rd sort key  */
+  cmp = strcmp (p1->name, p2->name);
+  if (cmp)
+    {
+      /* Two different opcodes  */
+      size_t l1 = strlen (p1->name);
+      size_t l2 = strlen (p2->name);
+      ptrdiff_t i1 = strchr (p1->name, '.') - p1->name;
+      ptrdiff_t i2 = strchr (p2->name, '.') - p2->name;
+      char c1, c2;
+
+      if ((l1 == l2) && (i1 == i2) && (i1 > 0))
+	{
+	  /* Check for different mnemonics of the same length,
+	     e.g. add.w vs. and.b  */
+	  cmp = strncmp (p1->name, p2->name, i1);
+	  if (cmp)
+	    return cmp;
+
+	  /* At this point we expect only b, w or l suffix,
+	     where b < w < l  */
+	  c1 = p1->name[i1+1];
+	  c2 = p2->name[i2+1];
+	  if (c1 == 'b')
+	    return -1;
+	  else if (c1 == 'w')
+	    return (c2 == 'b') ? 1 : -1;
+	  else
+	    return 1;
+	}
+
+      return cmp;
+    }
+
+  /* The 4th sort key  */
+  return p1->how - p2->how;
 }
 
 
-- 
2.55.0

Re: [REPOST][PATCH v4] H8/300: sim: Fix simulator hang caused by qsort on Windows/MinGW

[Jeff Law]

On 10/5/26 12:05 AM, Jan Dubiec wrote:
> On Windows, qsort() uses an unstable sorting algorithm, which results
> in a "shuffled" opcode table rather than a properly sorted one, causing
> the entire simulator to hang.
>
> The simulator happens to work on Linux, but this behavior is not
> guaranteed, because the glibc documentation clearly states that "If
> two elements compare equal, their order after sorting is unpredictable."
>
> This patch introduces two additional sort keys to the instruction
> comparator function, making the resulting opcode table as close as
> possible to the one that would be produced by a stable sorting algorithm.
>
> It also fixes an issue in the instruction decoder where, for example,
> mov.l @er7+,er1 is recognized as ldm.l @er7+,(er0-er1) if the opcode
> table is sorted "the wrong way", i.e. when ldm appears before mov
> in the table.
>
> No regressions were found.
This seems generally sensible to me.  I'd probably kill that ldm/stm 
special case, particularly if the new comparator avoids the problem.

I'd instead be more interested in tripping an assert if we ever have two 
things compare equal


jeff

Re: [REPOST][PATCH v4] H8/300: sim: Fix simulator hang caused by qsort on Windows/MinGW

[Jan Dubiec]

On 8.10.2026 04:04, Jeff Law wrote:
[...]

> This seems generally sensible to me.  I'd probably kill that ldm/stm 
> special case, particularly if the new comparator avoids the problem.

I'm not sure what you mean by "new". That said, using a stable sorting 
algorithm avoids the problem, but only incidentally.

The root of the problem is that the third nibble of the LDM/STM opcode 
is marked as DATA in h8_opcodes[]. As a result, if h8_opcodes[] is 
sorted in the "wrong" order, decode() may decode MOV.L @ERs+, ERd as 
LDM/STM. The first three bytes of the MOV.L instruction are 01 00 6D, 
whereas those of LDM/STM are 01 X0 6D, where X is 1, 2, or 3. That's why 
I added the special case.

IMO, the LDM/STM definition should be split into three separate 
instructions, as specified in the "H8S/2600 Series, H8S/2000 Series 
Software Manual". However, I didn't want to modify h8_opcodes[] because 
it is also used by other tools.

Also, the simulator is slightly slower when using a stable sorting 
algorithm.

> I'd instead be more interested in tripping an assert if we ever have two 
> things compare equal
An assert doesn't make sense here because equal comparisons occur "all 
the time". For H8/300H, the comparator is called 80919 times and returns 
zero 22434 times. For H8S, the corresponding numbers are 94315/26963, 
and for H8SX, 79552/23240. The h8_opcodes[] array contains 8484 elements.

/J.D.

Re: [REPOST][PATCH v4] H8/300: sim: Fix simulator hang caused by qsort on Windows/MinGW

[Jeff Law]

On 10/8/26 9:14 PM, Jan Dubiec wrote:
> On 8.10.2026 04:04, Jeff Law wrote:
> [...]
>
>> This seems generally sensible to me.  I'd probably kill that ldm/stm 
>> special case, particularly if the new comparator avoids the problem.
>
> I'm not sure what you mean by "new". That said, using a stable sorting 
> algorithm avoids the problem, but only incidentally.
Then we probably need to further improve the stability of the sort.

>
>> I'd instead be more interested in tripping an assert if we ever have 
>> two things compare equal
> An assert doesn't make sense here because equal comparisons occur "all 
> the time". For H8/300H, the comparator is called 80919 times and 
> returns zero 22434 times. For H8S, the corresponding numbers are 
> 94315/26963, and for H8SX, 79552/23240. The h8_opcodes[] array 
> contains 8484 elements.
Those would all seem like problems waiting to happen.  If qsort ever 
returns 0, then you've got the potential for an unstable sort.

Jeff