trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto van der Schaaf <osch...@gmail.com>
Subject Re: Api Questions
Date Fri, 25 Jan 2013 20:34:36 GMT
I hope that works out, might be helpful and prevent other from having
to go down the same road :-) I would be happy to help out with that.

Regarding INKVIONBytesGet:

"Gets the number of bytes to be performed by the IO operation
described by viop. This is the nbytes parameter that's passed to
INKVConnRead or INKVConnWrite."

So, no, I don't think that modifying anything. If I remember
correctly, that call returns the total number of bytes that the will
be written to the passed in vio. So in the context of the picture
above, you will write as much bytes to the output as you get from the
input I guess. You can specify INT64_MAX if you don't know yet how
much bytes you will write, in which case trafficserver will either
send a connection:close header or send a transfer-encoding chunked.

You might want to have a look at the gzip plugin in /experimental: I
made an effort to make to code a little more understandable then the
null-transform, and I think it's also a little more experimental :-)


Regards,

Otto


2013/1/25 Owens, Steve <Steve.Owens@disney.com>:
> Otto,
>
> I have started mapping code examples from the null transform example to
> the diagram to try to connect the dots see attached.
>
> Steve Owens
>
>
>
> On 1/25/13 11:44 AM, "Owens, Steve" <Steve.Owens@disney.com> wrote:
>
>>Otto, yes I have seen them and read them several times.  The second link
>>explains in rather abstract terms what VIO's do and how they relate to
>>buffers.
>>
>>But I personally could use a little more amplification on what the
>>specific methods are that manipulate these structures and what effect that
>>calls to these methods have on the structures depicted in the images.
>>
>>The null transform plugin re-uses variables and takes enough shortcuts
>>that what it is doing is seems obscured and I am trying to pierce that
>>obscurity.  I will continue to try to pierce the veil but after having
>>done so I would love to write up additional info to make it clearer to
>>dunderheads such as myself what these methods actually do in order to
>>reduce the slope of the learning curve for this technology.
>>
>>Thank you for re-directing me to these I am sure that given enough time
>>and patience I will crack the code.
>>
>>
>>
>>On 1/25/13 11:25 AM, "Otto van der Schaaf" <oschaaf@gmail.com> wrote:
>>
>>>Hi Steve,
>>>
>>>Did you get a chance already to have a look at
>>>http://trafficserver.apache.org/docs/v2/sdk/images/vconnection.jpg ?
>>>That image is from
>>>http://trafficserver.apache.org/docs/v2/sdk/HTTPTransformationPlugins.htm
>>>l
>>>#VIOs
>>>Both helped my understanding of the transformation api's
>>>
>>>Regards,
>>>
>>>Otto
>>>
>>>
>>>2013/1/25 Owens, Steve <Steve.Owens@disney.com>:
>>>> One step closer
>>>>
>>>> VIO *
>>>>
>>>> PluginVC::do_io_write(Continuation * c, int64_t nbytes, IOBufferReader
>>>>*
>>>> abuffer, bool owner)
>>>>
>>>> {
>>>>
>>>>
>>>>   ink_assert(!closed);
>>>>
>>>>   ink_assert(magic == PLUGIN_VC_MAGIC_ALIVE);
>>>>
>>>>
>>>>   if (abuffer) {
>>>>
>>>>     ink_assert(!owner);
>>>>
>>>>     write_state.vio.buffer.reader_for(abuffer);
>>>>
>>>>   } else {
>>>>
>>>>     write_state.vio.buffer.clear();
>>>>
>>>>   }
>>>>
>>>>
>>>>   // Note: we set vio.op last because process_write_side looks at it to
>>>>
>>>>   //  tell if the VConnection is active.
>>>>
>>>>   write_state.vio.mutex = c->mutex;
>>>>
>>>>   write_state.vio._cont = c;
>>>>
>>>>   write_state.vio.nbytes = nbytes;
>>>>
>>>>   write_state.vio.ndone = 0;
>>>>
>>>>   write_state.vio.vc_server = (VConnection *) this;
>>>>
>>>>   write_state.vio.op = VIO::WRITE;
>>>>
>>>>
>>>>   Debug("pvc", "[%u] %s: do_io_write for %"PRId64" bytes", PVC_ID,
>>>>PVC_TYPE,
>>>> nbytes);
>>>>
>>>>
>>>>   // Since reentrant callbacks are not allowed on from do_io
>>>>
>>>>   //   functions schedule ourselves get on a different stack
>>>>
>>>>   need_write_process = true;
>>>>
>>>>   setup_event_cb(0, &sm_lock_retry_event);
>>>>
>>>>
>>>>   return &write_state.vio;
>>>>
>>>> }
>>>>
>>>>
>>>> It would appear that the value passed in to the third argument of
>>>> TSVConnWrite ends up being assigned to write_state.vio.nbytes
>>>>
>>>> the docs on write_state.vio.nbytes state:
>>>>
>>>> /**
>>>>
>>>>     Number of bytes to be done for this operation.
>>>>
>>>>
>>>>     The total number of bytes this operation must complete.
>>>>
>>>>
>>>>   */
>>>>
>>>>   int64_t nbytes;
>>>>
>>>>
>>>> So far clear as mud.  Apologies in advance for my mental density.
>>>>
>>>>
>>>>
>>>> From: "Owens, Steve" <Steve.Owens@disney.com>
>>>> Reply-To: "users@trafficserver.apache.org"
>>>><users@trafficserver.apache.org>
>>>> Date: Fri, 25 Jan 2013 11:09:08 -0800
>>>> To: "users@trafficserver.apache.org" <users@trafficserver.apache.org>
>>>> Subject: Re: Api Questions
>>>>
>>>> An addendum to the previous message I have resorted to reading the
>>>>traffic
>>>> server code in order to try to discern what it does:
>>>>
>>>> I am examining
>>>>
>>>> TSVIO
>>>>
>>>> TSVConnWrite(TSVConn connp, TSCont contp, TSIOBufferReader readerp,
>>>>int64_t
>>>> nbytes)
>>>>
>>>> {
>>>>
>>>>   sdk_assert(sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS);
>>>>
>>>>   sdk_assert(sdk_sanity_check_iocore_structure(contp) == TS_SUCCESS);
>>>>
>>>>   sdk_assert(sdk_sanity_check_iocore_structure(readerp) == TS_SUCCESS);
>>>>
>>>>   sdk_assert(nbytes >= 0);
>>>>
>>>>
>>>>   FORCE_PLUGIN_MUTEX(contp);
>>>>
>>>>   VConnection *vc = (VConnection *) connp;
>>>>
>>>>
>>>>   return reinterpret_cast<TSVIO>(vc->do_io_write((INKContInternal
*)
>>>>contp,
>>>> nbytes, (IOBufferReader *) readerp));
>>>>
>>>> }
>>>>
>>>>
>>>> It would appear that the answer to what this method does lies in what
>>>> happens in the function "vc->do_io_write" yet when I search the
>>>> trafficserver 3.2.0 codebase for any file that actually contains an
>>>> implementation of "vc->do_io_write" I find that "do_io_write" is
>>>>declared in
>>>> several .h files but not defined in any .c or .cpp files.
>>>>
>>>>
>>>> Does anyone know where in the code base the implementation of
>>>>"do_io_write"
>>>> resides?
>>>>
>>>>
>>>>
>>>>
>>>> From: "Owens, Steve" <Steve.Owens@disney.com>
>>>> Reply-To: "users@trafficserver.apache.org"
>>>><users@trafficserver.apache.org>
>>>> Date: Fri, 25 Jan 2013 10:48:11 -0800
>>>> To: "users@trafficserver.apache.org" <users@trafficserver.apache.org>
>>>> Subject: Api Questions
>>>>
>>>> I am trying to understand the relationship between the following
>>>>functions
>>>> as they are used within a transformation plugin and so far no matter
>>>>how
>>>> hard I dig the mystery remains.  I am hoping that people who have a
>>>>better
>>>> understanding of ATS than I do would be willing to help clear up any
>>>> misunderstandings I have.
>>>>
>>>> Here are the functions I find confusing:
>>>>
>>>> TSVIONBytesSet
>>>>
>>>> (TSIOBufferCopy and or TSIOBufferWrite)
>>>>
>>>> And finally
>>>>
>>>> TSVConnWrite
>>>>
>>>>
>>>> Each of the above functions takes an int_64 argument at the end to
>>>>indicate
>>>> the number of bytes but I don't fully understand the purpose of this
>>>> parameter with respect to each of the functions.
>>>>
>>>>
>>>> Suppose I have a transformation which is intended to write a prefix
>>>>spew the
>>>> response and then write a suffix to the down stream I would imagine it
>>>>looks
>>>> something like this:
>>>>
>>>>
>>>>
>>>>
>>>> void handleTransformOperation(TSCont contp) {
>>>>
>>>> TSIOBuffer buf_test;
>>>>
>>>> int_64 written = 0;
>>>>
>>>> TransormationData* pData = (TransformationData*) TSContDataGet(contp);
>>>>
>>>> TSVIO input_vio = TSTransformOutputVConnGet(contp);
>>>>
>>>>
>>>> if(data->state == STATE_BEGIN) {
>>>>
>>>>     // Initialize the output_buffer and output_reader
>>>>
>>>>     data->output_buffer = TSIOBufferCreate();
>>>>
>>>>     data->output_reader = TSIOBufferReaderAlloc(data->output_buffer);
>>>>
>>>>
>>>>
>>>>    // Write out a prefix
>>>>
>>>>    written += TSIOBufferWrite(data->output_buffer, "{Hello I am a
>>>>prefix}",
>>>> strlen(prefix) * sizeof(char));
>>>>
>>>>    data->state == STATE_MIDDLE;
>>>>
>>>>
>>>>    // Figure out if the upstream wants to stop receiving WRITEREADY and
>>>> WRITE_COMPLETE events
>>>>
>>>>    buf_test = TSVIOBufferGet(input_vio);
>>>>
>>>>    if (!buf_test) {
>>>>
>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>
>>>>        TSVIONBytesSet(data->output_vio, /*What goes here doneget or
>>>>written
>>>> + doneget or just written?  Bonus question: why? /* ???);
>>>>
>>>>        data->state = STATE_END;
>>>>
>>>>    } else
>>>>
>>>>        data->state = STATE_MIDDLE;
>>>>
>>>> }
>>>>
>>>>
>>>> // Notice that the above if falls through to the next state so it is
>>>> possible that the following code may or may not
>>>>
>>>> // get executed based on the buf_test result above.
>>>>
>>>> if(data->state == STATE_MIDDLE) {
>>>>
>>>>    // Figure out if the upstream wants to stop receiving WRITEREADY and
>>>> WRITE_COMPLETE events
>>>>
>>>>    buf_test = TSVIOBufferGet(input_vio);
>>>>
>>>>    if (!buf_test) {
>>>>
>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>
>>>>        TSVIONBytesSet(data->output_vio, /*What goes here doneget or
>>>>written
>>>> + doneget or just written?  Bonus question: why? /* ???);
>>>>
>>>>        data->state = STATE_END;
>>>>
>>>>    } else {
>>>>
>>>>        // Figure out how much data is left to read from the upstream
>>>>
>>>>        int64_t upstreamBytesRemaining = TSVIONTodoGet(input_vio);
>>>>
>>>>        int64_t upstreamBytesAvail =
>>>> TSIOBufferReaderAvail(TSVIOReaderGet(input_vio));
>>>>
>>>>        if(upstreamBytesRemaining > 0) {
>>>>
>>>>             int64_t bytesToWrite = upstreamBytesRemaining;
>>>>
>>>>             if(bytesToWrite > upstreamBytesAvail)
>>>>
>>>>                  bytesToWrite = upstreamBytesAvail;
>>>>
>>>>
>>>>
>>>>             /* Copy the data from the read buffer to the output buffer.
>>>>*/
>>>>
>>>>     TSIOBufferCopy(TSVIOBufferGet(data->output_vio),
>>>> TSVIOReaderGet(input_vio), bytesToWrite, 0);
>>>>
>>>>
>>>>            /* Tell the read buffer that we have read the data and are
>>>>no
>>>>
>>>>     * longer interested in it.
>>>>
>>>>     */
>>>>
>>>>     TSIOBufferReaderConsume(TSVIOReaderGet(input_vio), bytesToWrite);
>>>>
>>>>
>>>>            /* Modify the input VIO to reflect how much data we've
>>>>
>>>>     * completed.
>>>>
>>>>     */
>>>>
>>>>            int64_t doneget = TSVIONDoneGet(input_vio);
>>>>
>>>>    TSVIONDoneSet(input_vio, /* This is where things get really muddy
>>>>what
>>>> goes here?  doneget or doneget + bytesToWrite or doneget + bytesToWrite
>>>>+
>>>> written */ ??);
>>>>
>>>>            /* What is the impact of calling this later in the handler
>>>>if it
>>>> has already been called here?
>>>>
>>>>        }
>>>>
>>>>
>>>>        /* Now we check the input VIO to see if there is data left to
>>>>
>>>> * read.
>>>>
>>>> */
>>>>
>>>>         upstreamBytesRemaining = TSVIONTodoGet(input_vio);
>>>>
>>>> if (upstreamBytesRemaining > 0) {
>>>>
>>>>     if (upstreamBytesAvail > 0) {
>>>>
>>>>       /* If there is data left to read, then we reenable the output
>>>>
>>>>        * connection by reenabling the output VIO. This will wake up
>>>>
>>>>        * the output connection and allow it to consume data from the
>>>>
>>>>        * output buffer.
>>>>
>>>>        */
>>>>
>>>>        TSVIOReenable(data->output_vio);
>>>>
>>>>
>>>>        /* Call back the input VIO continuation to let it know that we
>>>>
>>>>         * are ready for more data.
>>>>
>>>>         */
>>>>
>>>>        TSContCall(TSVIOContGet(input_vio),
>>>>
>>>> TS_EVENT_VCONN_WRITE_READY, input_vio);
>>>>
>>>>     }
>>>>
>>>> } else {
>>>>
>>>>     /* There is no data left to read so set state to STATE_END and */
>>>>
>>>>     data->state = STATE_END;
>>>>
>>>> }
>>>>
>>>>    }
>>>>
>>>>
>>>>    // The above can fall through to this if block
>>>>
>>>>    if(data->state == STATE_END) {
>>>>
>>>>        /* Spew a suffix */
>>>>
>>>>        const char* suffix = "{i am a suffix}";
>>>>
>>>>        int64_t localWritten = TSIOBufferWrite(data->output_buffer,
>>>>suffix,
>>>> (strlen(suffix) - 1) * sizeof(char));
>>>>
>>>>
>>>>
>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>
>>>>        TSVIONBytesSet(data->output_vio, /* What goes here? Why? */??);
>>>>
>>>>        TSVIOReenable(data->output_vio);
>>>>
>>>>
>>>>       /* Call back the input VIO continuation to let it know that we
>>>>
>>>>        * have completed the write operation.
>>>>
>>>>        */
>>>>
>>>>        TSContCall(TSVIOContGet(input_vio),
>>>>TS_EVENT_VCONN_WRITE_COMPLETE,
>>>> input_vio);
>>>>
>>>>    }
>>>>
>>>>
>>>>    /* I am presuming we only want to call this once per invocation of
>>>> handlerOperation  But I am not sure */
>>>>
>>>>   data->output_vio = TSVConnWrite(output_conn, contp,
>>>>data->output_reader,
>>>> /* What to put here hmmm and why? */ ?? );
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> I really want to understand how these methods relate and what they do.
>>>>I
>>>> would be willing to help update/contribute to the ATS docs but first I
>>>>would
>>>> need to understand what they do and so far the examples that are
>>>>available
>>>> have not made it clear to me what is going on as yet so any help would
>>>>be
>>>> appreciated.
>>>>
>>>>
>>>>
>>>>
>>
>

Mime
View raw message