trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shaun mcginnity <shaun.mcginn...@gmail.com>
Subject Re: API Question
Date Fri, 15 Mar 2013 23:35:08 GMT
Thanks Steve, glad to help.

Shaun

On Friday, 15 March 2013, Owens, Steve wrote:

> Shaun,
>
> Thanks to your input I was able to identify the source of the problem.
>
> In my main handle transformation method I was declaring a local variable  int64_t
> nBytesWritten = 0;
>
> Then depending on what state I was in I was calling the appropriate
> handler sub method.  Each of the sub methods returns the number of bytes
> written to the output by that method and I was aggregating those values via
>
> nBytesWritten += subMethodCall(…);
>
> At the end of my handle transformation method I was calling:
>
> if (nBytesWritten > 0) {
>
>                 /* Modify the output
>
>                  * VIO to reflect how much data the output connection
> should
>
>                  * expect. This allows the output connection to know when
> it
>
>                  * is done reading. We then reenable the output connection
> so
>
>                  * that it can consume the data we just gave it.
>
>                  */
>
>                 TSVIONBytesSet(data->output_vio, nBytesWritten);
>
>                 TSVIOReenable(data->output_vio);
>
> }
>
>
>
> At any rate this was working for small payloads but when the method was
> re-entered nBytesWritten was being re-initialized to 0 and then
> incremented.  *So the key is knowing that the value of nBytesWritten must
> be accumulated across invocations of the transformation handler*.
>
>
> by making this value part of the transaction data, I am able to preserve
> it across the invocations and the modified code:
>
> if (data->nBytesWritten > 0) {
>
>                 /* Modify the output
>
>                  * VIO to reflect how much data the output connection
> should
>
>                  * expect. This allows the output connection to know when
> it
>
>                  * is done reading. We then reenable the output connection
> so
>
>                  * that it can consume the data we just gave it.
>
>                  */
>
>                 TSVIONBytesSet(data->output_vio, data->nBytesWritten);
>
>                 TSVIOReenable(data->output_vio);
>
> }
>
>
> I am worried though about what you said regarding it getting confused and
> sending a WRITE_COMPLETE event pre-maturely.  Right now with my current
> test cases seem to be working as expected but then again, it all seems like
> voodoo and black magic under the ATS hood and I don't know that there isn't
> a test case which would break this.  I wish I could be more confident that
> the code I am writing is going to work as intended under all cases.
>
> For the moment however everything seems to work.  I am logging the values
> of the output nbytes and ndone.  And it would seem that ndone is not
> getting incremented at all during the multiple invocations of my transform
> handler.  Ideally what I would like to do is to have the down stream output
> handler consume data as soon as it gets it from my transform, but not to
> stop or terminate until I have finished sending ALL of my data downstream
> but I am afraid to breathe on the code at this point until I find a
> breaking case which will mandate a change.
>
> Thank you for your help, you have truly made my day.
>
> Steve Owens
>
>
> From: Shaun mcginnity <shaun.mcginnity@gmail.com <javascript:_e({},
> 'cvml', 'shaun.mcginnity@gmail.com');>>
> Reply-To: "users@trafficserver.apache.org <javascript:_e({}, 'cvml',
> 'users@trafficserver.apache.org');>" <users@trafficserver.apache.org<javascript:_e({},
'cvml', 'users@trafficserver.apache.org');>
> >
> Date: Fri, 15 Mar 2013 10:35:01 -0700
> To: "users@trafficserver.apache.org <javascript:_e({}, 'cvml',
> 'users@trafficserver.apache.org');>" <users@trafficserver.apache.org<javascript:_e({},
'cvml', 'users@trafficserver.apache.org');>
> >
> Subject: Re: API Question
>
> Yes, it allows the VIO to know when it is done. We only set it when we
> have read all the data from the input. I suspect if you continually set it
> the VIO might think it is done and send a WRITE_COMPLETE but you actually
> still have more to write.
>
> Shaun
>
>
> On Fri, Mar 15, 2013 at 5:20 PM, Owens, Steve <Steve.Owens@disney.com<javascript:_e({},
'cvml', 'Steve.Owens@disney.com');>
> > wrote:
>
>> Shaun,
>>
>> Looking at the code for TSVIONBytesSet we have
>>
>> *void*281 <http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#281>TSVIONBytesSet
<http://code.metager.de/source/s?refs=TSVIONBytesSet&project=apache>(TSVIO <http://code.metager.de/source/s?defs=TSVIO&project=apache>
viop <http://code.metager.de/source/s?refs=viop&project=apache>, int64_t <http://code.metager.de/source/s?defs=int64_t&project=apache>
nbytes <http://code.metager.de/source/s?refs=nbytes&project=apache>)282 <http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#282>{283
<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#283>
 sdk_assert <http://code.metager.de/source/s?defs=sdk_assert&project=apache>(sdk_sanity_check_iocore_structure
<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#sdk_sanity_check_iocore_structure>(viop
<http://code.metager.de/source/s?defs=viop&project=apache>) == TS_SUCCESS <http://code.metager.de/source/s?defs=TS_SUCCESS&project=apache>);284
<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#284>
 sdk_assert <http://code.metager.de/source/s?defs=sdk_assert&project=apache>(nbytes
<http://code.metager.de/source/s?defs=nbytes&project=apache> >= 0);285 <http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#285>286
<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#286>
 VIO <http://code.metager.de/source/s?defs=VIO&project=apache> *vio <http://code.metager.de/source/s?refs=vio&project=apache>
= (VIO <http://code.metager.de/source/s?defs=VIO&project=apache> *)viop <http://code.metager.de/source/s?defs=viop&project=apache>;287
<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#287>
 vio <http://code.metager.de/source/s?defs=vio&project=apache>->nbytes <http://code.metager.de/source/s?defs=nbytes&project=apache>
= nbytes <http://code.metager.de/source/s?defs=nbytes&project=apache>;288 <http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#288>}
>>
>> It looks like this method updates the nbytes of the VIO.  But it doesn't explain
what the implications of changing this value are.  I am presuming that when I called TSVConnWrite(output_conn,
contp, data->output_reader, INT64_MAX);  That the value of vio->nbytes got set to INT64_MAX.
 Since my transform may or may not know how much upstream data it needs to process I don't
see any way for it to update this value to anything meaningful and it makes me wonder why
I would want to.
>>
>> In my STATE_EXIT processing I am calling: TSContCall(TSVIOContGet(input_vio), TS_EVENT_VCONN_WRITE_COMPLETE,
input_vio); But I don't call: TSVConnShutdown(TSTransformOutputVConnGet(contp), 0, 1); until
I receive the TS_EVENT_VCONN_WRITE_COMPLETE event in my callbackHandler.
>>
>> What would happen if I continuously update TSVIONBytesSet(output_vio, total_written)
to indicate how much I have written thus far?
>> Steve Owens
>>
>>
>> From: Shaun mcginnity <shaun.mcginnity@gmail.com>
>> Reply-To: "users@trafficserver.apache.org" <
>> users@trafficserver.apache.org>
>> Date: Fri, 15 Mar 2013 09:48:05 -0700
>>
>> To: "users@trafficserver.apache.org" <users@trafficserver.apache.org>
>> Subject: Re: API Question
>>
>> Hi Steve,
>>
>> I believe that in a transform you should only call TSVConnWrite once.
>>
>> You say you are shutting down the VIOs. Are you doing this when you
>> receive a WRITE_COMPLETE event? If not you may be closing the VIO early.
>> You can tell the output VIO that all the data has been written to it using
>> TSVIONBytesSet(output_vio, total_written);
>>
>> Regards,
>>
>> Shaun
>>
>>
>> On Fri, Mar 15, 2013 at 4:25 PM, Owens, Steve <Steve.Owens@disney.com>wrote:
>>
>> Shaun,
>>
>> This is extremely helpful.  In fact it would make a great addition to the
>> traffic server on line documentation.  Personally I prefer to understand
>> what the methods I am calling do rather than not.
>>
>> The one question that remains unanswered is whether or not this method
>> needs to be called more than once.
>>
>> The reason for my question is this:
>>
>> I have a transform which essentially prepends and postpends content to
>> the upstream content being served.  It works in states
>>
>> STATE_START = write the prefix content downstream and shift to
>> STATE_MIDDLE.
>>         STATE_MIDDLE = copy upstream content to downstream until there is
>> no more content and then shift to STATE_END.
>>         STATE_END = write the suffix content downstream and shift to
>> STATE_EXIT.
>>         STATE_EXIT = clean up the transaction and shut down the VIO's
>> because were done.
>>
>> This plugin works with smaller upstream content as expected.  However
>> when the upstream content gets larger than about 30K, what I am seeing is
>> that from the transform plugin's perspective it is working as expected.
>>  The expected amount of data is being copied downstream and the suffix is
>> being written.  I can tell this because I am logging the values of
>> upstream_avail, upstream_todo, bytesWritten etc.  And everything adds up to
>> all content being delivered.  However the client is receiving truncated
>> content and I am baffled as to why this my be happening.  I thought that
>> maybe I was not properly using the TSVConnWrite method but from your
>> explanation of the method below, it would seem that this is possibly not
>> the case.
>>
>> At any rate,
>>
>> Thank you for the explanation.
>>
>> Best Regards,
>>
>> Steve Owens
>>
>> From: Shaun mcginnity <shaun.mcginnity@gmail.com>
>> Reply-To: "users@trafficserver.apache.org" <users
>>
>>
>

Mime
View raw message