Re: [PATCH 3/6] gdbsupport: use dynamic partitioning in gdb::parallel_for_each

[Simon Marchi <simark@simark.ca>]

On 2025-06-13 16:12, Simon Marchi wrote:
> On 6/13/25 3:56 PM, Simon Marchi wrote:
>> On 6/13/25 3:22 PM, Simon Marchi wrote:
>>> Another idea I had (but didn't try) was to make lambdas receive a
>>> "magic" range object, like athis:
>>>
>>>   [&] (gdb::dynamic_range<dwarf2_per_cu *> range)
>>>     {
>>>       for (dwarf2_per_cu *cu : range)
>>>         process_unit (cu);
>>>     }
>>>
>>> "range" would get work items from the work queue in batches, but yield
>>> one at a time.  It would reach its end when the work queue is empty.
>>> It will be a bit of work to implement, but it would have the advantage
>>> that any per-worker state could be right there in the lambda, like we
>>> have today.
>>
>> Ah, I remembered why I didn't do this.  It wouldn't play well with how
>> minimal_symbol_reader::install() currently uses gdb::parallel_for_each.
>> Currently, it receives one contiguous range [start,end).  After having
>> computed the demangled names and hashes for the whole range, each worker
>> locks a mutex and installs the names in a shared hash table.
>>
>> In my patch, it's similar, but each worker installs the names in the
>> share hash table at the end of each small range it receives.
>>
>> With my "magic" iterator idea, it wouldn't be clear when and how to
>> install the names in the shared hash table.
> 
> One way to make this work would be to make the iterator yield batches,
> so you would have two levels of iteration:
> 
>   [&] (gdb::dynamic_range<minimal_symbol *, 1000> range)
>     {
>       for (gdb::batch<minimal_symbol *> batch : range)
>         {
> 	  for (minimal_symbol *msym : batch)
> 	    {
> 	      // compute demangled name and hash
> 	    }
> 
> 	  // lock mutex
> 
> 	  for (minimal_symbol *msym : batch)
> 	    {
> 	      // install msym in the shared hash table
> 	    }
> 	}
>     }
> 
> Let me know if you see an easier solution to this.

I have thought about this problem some more, and I discussed this with
my team.  I still think that the functor approach is the only one that
relatively cleanly meets the various requirements:

 - be able to run a per-worker initialize step
 - be able to keep some per-worker data for the duration of the for-each
 - be able to run a per-worker finalize step

Simon