[lttng-dev] Overwrite mode, lost events, and reader behaviour

[lttng-dev] Overwrite mode, lost events, and reader behaviour

[Thibault, Daniel]

   I'd like a clarification.  In overwrite mode, and assuming the writers are overwhelming the consumers, what is the behaviour?

   Assume we have this situation within a given channel (numbering the sub-buffers in chronological order), and the writers are about to catch up with the reader (the extra sub-buffer is double-bracketed and the sub-buffer it is "shadowing" is marked with a trailing asterisk):

[8] [9] [2] [3] [4] [5] [6] [7] [[-]]   reader at [2], writers at [9], spare sub-buffer unused/unallocated

[8] [9] [10]* [3] [4] [5] [6] [7] [[2]]   reader at [[2]], writers at [10], spare sub-buffer shadowing [10]

[8] [9] [10]* [11] [4] [5] [6] [7] [[2]]   reader at [[2]], writers at [11], spare sub-buffer shadowing [10]

   Now, when the reader completes evacuating [[2]], it moves on to [11] and realises there is a time gap.  I suspect it does not just jump its clock ahead, because that would give up on [4] through [10].  Does it skip to the next sub-buffer until it sees the clock go backward (the circularity of the buffer guarantees there is always one such discontinuity), resuming its work with [4]?  If yes, in the worst case the reader will go once around the sub-buffers, checking timestamps, before resuming its work.  This could be avoided if the buffer metadata kept the spare sub-buffer chained to the oldest one (i.e. the third step is [8] [9] [10] [11]* [4] [5] [6] [7] [[2]]), so the reader never needs to call "next()" more than once.  Is this what lttng already does?

Daniel U. Thibault
Protection des syst?mes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber s?curit? pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la d?fense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Qu?bec QC? G3J 1X5
CANADA
Vox?: (418) 844-4000 x4245
Fax?: (418) 844-4538
NAC?: 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada?| Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

[lttng-dev] Overwrite mode, lost events, and reader behaviour

[Mathieu Desnoyers]

----- Original Message -----
> From: "Daniel Thibault" <Daniel.Thibault@drdc-rddc.gc.ca>
> To: lttng-dev at lists.lttng.org
> Sent: Monday, November 18, 2013 9:43:06 AM
> Subject: [lttng-dev] Overwrite mode, lost events, and reader behaviour
> 
> I'd like a clarification.  In overwrite mode, and assuming the writers are
> overwhelming the consumers, what is the behaviour?
> 
>    Assume we have this situation within a given channel (numbering the
>    sub-buffers in chronological order), and the writers are about to catch
>    up with the reader (the extra sub-buffer is double-bracketed and the
>    sub-buffer it is "shadowing" is marked with a trailing asterisk):
> 
> [8] [9] [2] [3] [4] [5] [6] [7] [[-]]   reader at [2], writers at [9], spare
> sub-buffer unused/unallocated
> 
> [8] [9] [10]* [3] [4] [5] [6] [7] [[2]]   reader at [[2]], writers at [10],
> spare sub-buffer shadowing [10]
> 
> [8] [9] [10]* [11] [4] [5] [6] [7] [[2]]   reader at [[2]], writers at [11],
> spare sub-buffer shadowing [10]
> 
>    Now, when the reader completes evacuating [[2]], it moves on to [11] and
>    realises there is a time gap.  I suspect it does not just jump its clock
>    ahead, because that would give up on [4] through [10].  Does it skip to
>    the next sub-buffer until it sees the clock go backward (the circularity
>    of the buffer guarantees there is always one such discontinuity),
>    resuming its work with [4]?

We don't use the timestamps for this. Each packet has a sequence number given by
the producer, and the reader is querying for packets by requesting sequence
numbers (increasing by one each time). In the case you present here, the reader
will try to query the sequence number of an overwritten packet, and will therefore
not be able to get it. It will move on to the next packet until it finally finds
the first packet which sequence number matches its current sequence number (or the
writer position).

Mathieu


>  If yes, in the worst case the reader will go
>    once around the sub-buffers, checking timestamps, before resuming its
>    work.  This could be avoided if the buffer metadata kept the spare
>    sub-buffer chained to the oldest one (i.e. the third step is [8] [9] [10]
>    [11]* [4] [5] [6] [7] [[2]]), so the reader never needs to call "next()"
>    more than once.  Is this what lttng already does?
> 
> Daniel U. Thibault
> Protection des syst?mes et contremesures (PSC) | Systems Protection &
> Countermeasures (SPC)
> Cyber s?curit? pour les missions essentielles (CME) | Mission Critical Cyber
> Security (MCCS)
> R & D pour la d?fense Canada - Valcartier (RDDC Valcartier) | Defence R&D
> Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Qu?bec QC? G3J 1X5
> CANADA
> Vox?: (418) 844-4000 x4245
> Fax?: (418) 844-4538
> NAC?: 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada?| Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Overwrite mode, lost events, and reader behaviour

[Thibault, Daniel]

-----Message d'origine-----
Envoy??: 20 novembre 2013 09:44

> We don't use the timestamps for this. Each packet has a sequence number given by the producer, and the reader is querying for packets by requesting sequence numbers
> (increasing by one each time). In the case you present here, the reader will try to query the sequence number of an overwritten packet, and will therefore not be able to get it.
> It will move on to the next packet until it finally finds the first packet which sequence number matches its current sequence number (or the writer position).

   So my original suggestion holds: performance could increase just a little bit (on the reader side) if the reader's request (lib_ring_buffer_get_subbuf, is it?) did not return -EAGAIN but rather the current "earliest available sequence number" it should next ask for.  In the (presumably rare) case where the reader has been passed by the writers by several sub-buffer sequence numbers' worth, this would allow the reader to skip ahead to the next likely-to-succeed sequence number request.  Does this make sense?

Daniel U. Thibault
Protection des syst?mes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber s?curit? pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la d?fense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Qu?bec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

[lttng-dev] Overwrite mode, lost events, and reader behaviour

[Mathieu Desnoyers]

----- Original Message -----
> From: "Daniel Thibault" <Daniel.Thibault@drdc-rddc.gc.ca>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> Cc: lttng-dev at lists.lttng.org
> Sent: Wednesday, November 20, 2013 10:38:28 AM
> Subject: RE: [lttng-dev] Overwrite mode, lost events, and reader behaviour
> 
> -----Message d'origine-----
> Envoy??: 20 novembre 2013 09:44
> 
> > We don't use the timestamps for this. Each packet has a sequence number
> > given by the producer, and the reader is querying for packets by
> > requesting sequence numbers
> > (increasing by one each time). In the case you present here, the reader
> > will try to query the sequence number of an overwritten packet, and will
> > therefore not be able to get it.
> > It will move on to the next packet until it finally finds the first packet
> > which sequence number matches its current sequence number (or the writer
> > position).
> 
>    So my original suggestion holds: performance could increase just a little
>    bit (on the reader side) if the reader's request
>    (lib_ring_buffer_get_subbuf, is it?) did not return -EAGAIN but rather
>    the current "earliest available sequence number" it should next ask for.
>    In the (presumably rare) case where the reader has been passed by the
>    writers by several sub-buffer sequence numbers' worth, this would allow
>    the reader to skip ahead to the next likely-to-succeed sequence number
>    request.  Does this make sense?

Skipping ahead like this could indeed be a good approach. We don't need to
change get_subbuf() to do this though. When the consumer gets a -EAGAIN,
it could use the snapshot feature to find the new consumer position. Same
effect, no change needed in modules nor UST.

Thanks,

Mathieu

> 
> Daniel U. Thibault
> Protection des syst?mes et contremesures (PSC) | Systems Protection &
> Countermeasures (SPC)
> Cyber s?curit? pour les missions essentielles (CME) | Mission Critical Cyber
> Security (MCCS)
> R & D pour la d?fense Canada - Valcartier (RDDC Valcartier) | Defence R&D
> Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Qu?bec QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada | Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com