trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owens, Steve" <Steve.Ow...@disney.com>
Subject Re: API Question
Date Fri, 15 Mar 2013 22:28:03 GMT
That is a very useful thing to know.  Thank you Uri :-)


From: Uri Shachar <ushachar@hotmail.com<mailto:ushachar@hotmail.com>>
Reply-To: "users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>" <users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>>
Date: Fri, 15 Mar 2013 15:03:12 -0700
To: "users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>" <users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>>
Subject: RE: API Question

As a general comment -- There's no need to manipulate the nbytes every time you write to the
output.
If you call the initial TSVConnWrite with INT64_MAX, you just need to set the nbytes to the
final amount once (So the vconn will know to send you a WRITE_COMPLETE when it reaches this
amount).

If you know the real payload size before setting up the vconn, just call TSVConnWrite with
the real size and save yourself the hassle of adjusting the NBytes....

      Cheers,
                Uri

________________________________
From: Steve.Owens@disney.com<mailto:Steve.Owens@disney.com>
To: users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>
Date: Fri, 15 Mar 2013 11:13:33 -0700
Subject: Re: API Question

Shaun,

Thanks to your input I was able to identify the source of the problem.

In my main handle transformation method I was declaring a local variable  int64_t nBytesWritten
= 0;

Then depending on what state I was in I was calling the appropriate handler sub method.  Each
of the sub methods returns the number of bytes written to the output by that method and I
was aggregating those values via

nBytesWritten += subMethodCall(…);

At the end of my handle transformation method I was calling:

if (nBytesWritten > 0) {

                /* Modify the output

                 * VIO to reflect how much data the output connection should

                 * expect. This allows the output connection to know when it

                 * is done reading. We then reenable the output connection so

                 * that it can consume the data we just gave it.

                 */

                TSVIONBytesSet(data->output_vio, nBytesWritten);

                TSVIOReenable(data->output_vio);

}



At any rate this was working for small payloads but when the method was re-entered nBytesWritten
was being re-initialized to 0 and then incremented.  So the key is knowing that the value
of nBytesWritten must be accumulated across invocations of the transformation handler.


by making this value part of the transaction data, I am able to preserve it across the invocations
and the modified code:

if (data->nBytesWritten > 0) {

                /* Modify the output

                 * VIO to reflect how much data the output connection should

                 * expect. This allows the output connection to know when it

                 * is done reading. We then reenable the output connection so

                 * that it can consume the data we just gave it.

                 */

                TSVIONBytesSet(data->output_vio, data->nBytesWritten);

                TSVIOReenable(data->output_vio);

}


I am worried though about what you said regarding it getting confused and sending a WRITE_COMPLETE
event pre-maturely.  Right now with my current test cases seem to be working as expected but
then again, it all seems like voodoo and black magic under the ATS hood and I don't know that
there isn't a test case which would break this.  I wish I could be more confident that the
code I am writing is going to work as intended under all cases.

For the moment however everything seems to work.  I am logging the values of the output nbytes
and ndone.  And it would seem that ndone is not getting incremented at all during the multiple
invocations of my transform handler.  Ideally what I would like to do is to have the down
stream output handler consume data as soon as it gets it from my transform, but not to stop
or terminate until I have finished sending ALL of my data downstream but I am afraid to breathe
on the code at this point until I find a breaking case which will mandate a change.

Thank you for your help, you have truly made my day.

Steve Owens


From: Shaun mcginnity <shaun.mcginnity@gmail.com<mailto:shaun.mcginnity@gmail.com>>
Reply-To: "users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>" <users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>>
Date: Fri, 15 Mar 2013 10:35:01 -0700
To: "users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>" <users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>>
Subject: Re: API Question

Yes, it allows the VIO to know when it is done. We only set it when we have read all the data
from the input. I suspect if you continually set it the VIO might think it is done and send
a WRITE_COMPLETE but you actually still have more to write.

Shaun


On Fri, Mar 15, 2013 at 5:20 PM, Owens, Steve <Steve.Owens@disney.com<mailto:Steve.Owens@disney.com>>
wrote:
Shaun,

Looking at the code for TSVIONBytesSet we have

void281<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#281>TSVIONBytesSet<http://code.metager.de/source/s?refs=TSVIONBytesSet&project=apache>(TSVIO<http://code.metager.de/source/s?defs=TSVIO&project=apache>
viop<http://code.metager.de/source/s?refs=viop&project=apache>, int64_t<http://code.metager.de/source/s?defs=int64_t&project=apache>
nbytes<http://code.metager.de/source/s?refs=nbytes&project=apache>)
282<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#282>{
283<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#283>
 sdk_assert<http://code.metager.de/source/s?defs=sdk_assert&project=apache>(sdk_sanity_check_iocore_structure<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#sdk_sanity_check_iocore_structure>(viop<http://code.metager.de/source/s?defs=viop&project=apache>)
== TS_SUCCESS<http://code.metager.de/source/s?defs=TS_SUCCESS&project=apache>);
284<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#284>
 sdk_assert<http://code.metager.de/source/s?defs=sdk_assert&project=apache>(nbytes<http://code.metager.de/source/s?defs=nbytes&project=apache>
>= 0);
285<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#285>286<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#286>
 VIO<http://code.metager.de/source/s?defs=VIO&project=apache> *vio<http://code.metager.de/source/s?refs=vio&project=apache>
= (VIO<http://code.metager.de/source/s?defs=VIO&project=apache> *)viop<http://code.metager.de/source/s?defs=viop&project=apache>;
287<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#287>
 vio<http://code.metager.de/source/s?defs=vio&project=apache>->nbytes<http://code.metager.de/source/s?defs=nbytes&project=apache>
= nbytes<http://code.metager.de/source/s?defs=nbytes&project=apache>;
288<http://code.metager.de/source/xref/apache/trafficserver/trafficserver/proxy/InkIOCoreAPI.cc#288>}

It looks like this method updates the nbytes of the VIO.  But it doesn't explain what the
implications of changing this value are.  I am presuming that when I called TSVConnWrite(output_conn,
contp, data->output_reader, INT64_MAX);  That the value of vio->nbytes got set to INT64_MAX.
 Since my transform may or may not know how much upstream data it needs to process I don't
see any way for it to update this value to anything meaningful and it makes me wonder why
I would want to.

In my STATE_EXIT processing I am calling: TSContCall(TSVIOContGet(input_vio), TS_EVENT_VCONN_WRITE_COMPLETE,
input_vio); But I don't call: TSVConnShutdown(TSTransformOutputVConnGet(contp), 0, 1); until
I receive the TS_EVENT_VCONN_WRITE_COMPLETE event in my callbackHandler.

What would happen if I continuously update TSVIONBytesSet(output_vio, total_written) to indicate
how much I have written thus far?
Steve Owens



From: Shaun mcginnity <shaun.mcginnity@gmail.com<mailto:shaun.mcginnity@gmail.com>>
Reply-To: "users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>" <users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>>
Date: Fri, 15 Mar 2013 09:48:05 -0700

To: "users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>" <users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>>
Subject: Re: API Question

Hi Steve,

I believe that in a transform you should only call TSVConnWrite once.

You say you are shutting down the VIOs. Are you doing this when you receive a WRITE_COMPLETE
event? If not you may be closing the VIO early. You can tell the output VIO that all the data
has been written to it using TSVIONBytesSet(output_vio, total_written);

Regards,

Shaun


On Fri, Mar 15, 2013 at 4:25 PM, Owens, Steve <Steve.Owens@disney.com<mailto:Steve.Owens@disney.com>>
wrote:
Shaun,

This is extremely helpful.  In fact it would make a great addition to the traffic server on
line documentation.  Personally I prefer to understand what the methods I am calling do rather
than not.

The one question that remains unanswered is whether or not this method needs to be called
more than once.

The reason for my question is this:

I have a transform which essentially prepends and postpends content to the upstream content
being served.  It works in states

STATE_START = write the prefix content downstream and shift to STATE_MIDDLE.
        STATE_MIDDLE = copy upstream content to downstream until there is no more content
and then shift to STATE_END.
        STATE_END = write the suffix content downstream and shift to STATE_EXIT.
        STATE_EXIT = clean up the transaction and shut down the VIO's because were done.

This plugin works with smaller upstream content as expected.  However when the upstream content
gets larger than about 30K, what I am seeing is that from the transform plugin's perspective
it is working as expected.  The expected amount of data is being copied downstream and the
suffix is being written.  I can tell this because I am logging the values of upstream_avail,
upstream_todo, bytesWritten etc.  And everything adds up to all content being delivered. 
However the client is receiving truncated content and I am baffled as to why this my be happening.
 I thought that maybe I was not properly using the TSVConnWrite method but from your explanation
of the method below, it would seem that this is possibly not the case.

At any rate,

Thank you for the explanation.

Best Regards,

Steve Owens

From: Shaun mcginnity <shaun.mcginnity@gmail.com<mailto:shaun.mcginnity@gmail.com>>
Reply-To: "users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>" <users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>>
Date: Fri, 15 Mar 2013 07:09:12 -0700
To: "users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>" <users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>>
Subject: Re: API Question

Hi Steve,
TSVConnWrite initiates a write operation on the transform's downstream VConnection. The last
parameter is the total amount of data you are going to write. INT64_MAX is effectively unlimited.
When you have data to write you get the VIO's buffer (TSVIOBufferGet(output_vio)), and copy
the data into it. Then the VIO is reenabled. Reenabling the VIO causes an IMMEDIATE event
to be sent to the VIO's VConnection. When the VConnection's event handler has finished with
the data it will call your continuation back. If it expects you to send more data it will
pass a WRITE_READY event. If you have given the VIO the total amount of data as set in TSVConnWrite
it will call you back with a WRITE_COMPLETE event. Thisbehaviour is the same as what the transform
should do to its input VIO.
Since the downstream VConnection is the client and / or cache, if you do a finite write, i.e.
not INT64_MAX, then the HTTP transaction will close when the limit is reached. Issuing another
write in this case will probably not work. You can do multiple writes on a NetVConnection
though, e.g. as provided by TSNetConnect.
Hope that helps.
Shaun

-----Original Message-----
From: James Peach [mailto:jpeach@apache.org]
Sent: Friday, March 15, 2013 3:12 AM
To: users@trafficserver.apache.org<mailto:users@trafficserver.apache.org>
Subject: Re: API Question

On 14/03/2013, at 5:13 PM, "Owens, Steve" <Steve.Owens@disney.com<mailto:Steve.Owens@disney.com>>
wrote:

> In looking at the following documentation: http://trafficserver.apache.org/docs/trunk/sdk/io-guide/transformations.en.html
>
> It would seem that the only mention of TSVConnWrite is to a single parameter method.
>
> Yet in my plugin I am using
>
> data->output_vio = TSVConnWrite(output_conn, contp, data->output_reader,
>                                 INT64_MAX);
>
> What has changed about this method?
>
> What does it actually do?

It schedules data to be written to the VConnection.

>
> Do I need to call it every time the plugin is re-enabled?  Or can I just call it once?

I don't know that I really understand the question, but if the length is INT64_MAX you would
only call it once. If you want you cancall it a multiple times with shorter lengths.

J



Mime
View raw message