trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto van der Schaaf <osch...@gmail.com>
Subject Re: Api Questions
Date Fri, 25 Jan 2013 20:37:27 GMT
re: "and I think it's also a little more experimental "

I meant to say: and I think it has a more realistic function

2013/1/25 Otto van der Schaaf <oschaaf@gmail.com>:
> I hope that works out, might be helpful and prevent other from having
> to go down the same road :-) I would be happy to help out with that.
>
> Regarding INKVIONBytesGet:
>
> "Gets the number of bytes to be performed by the IO operation
> described by viop. This is the nbytes parameter that's passed to
> INKVConnRead or INKVConnWrite."
>
> So, no, I don't think that modifying anything. If I remember
> correctly, that call returns the total number of bytes that the will
> be written to the passed in vio. So in the context of the picture
> above, you will write as much bytes to the output as you get from the
> input I guess. You can specify INT64_MAX if you don't know yet how
> much bytes you will write, in which case trafficserver will either
> send a connection:close header or send a transfer-encoding chunked.
>
> You might want to have a look at the gzip plugin in /experimental: I
> made an effort to make to code a little more understandable then the
> null-transform, and I think it's also a little more experimental :-)
>
>
> Regards,
>
> Otto
>
>
> 2013/1/25 Owens, Steve <Steve.Owens@disney.com>:
>> Otto,
>>
>> I have started mapping code examples from the null transform example to
>> the diagram to try to connect the dots see attached.
>>
>> Steve Owens
>>
>>
>>
>> On 1/25/13 11:44 AM, "Owens, Steve" <Steve.Owens@disney.com> wrote:
>>
>>>Otto, yes I have seen them and read them several times.  The second link
>>>explains in rather abstract terms what VIO's do and how they relate to
>>>buffers.
>>>
>>>But I personally could use a little more amplification on what the
>>>specific methods are that manipulate these structures and what effect that
>>>calls to these methods have on the structures depicted in the images.
>>>
>>>The null transform plugin re-uses variables and takes enough shortcuts
>>>that what it is doing is seems obscured and I am trying to pierce that
>>>obscurity.  I will continue to try to pierce the veil but after having
>>>done so I would love to write up additional info to make it clearer to
>>>dunderheads such as myself what these methods actually do in order to
>>>reduce the slope of the learning curve for this technology.
>>>
>>>Thank you for re-directing me to these I am sure that given enough time
>>>and patience I will crack the code.
>>>
>>>
>>>
>>>On 1/25/13 11:25 AM, "Otto van der Schaaf" <oschaaf@gmail.com> wrote:
>>>
>>>>Hi Steve,
>>>>
>>>>Did you get a chance already to have a look at
>>>>http://trafficserver.apache.org/docs/v2/sdk/images/vconnection.jpg ?
>>>>That image is from
>>>>http://trafficserver.apache.org/docs/v2/sdk/HTTPTransformationPlugins.htm
>>>>l
>>>>#VIOs
>>>>Both helped my understanding of the transformation api's
>>>>
>>>>Regards,
>>>>
>>>>Otto
>>>>
>>>>
>>>>2013/1/25 Owens, Steve <Steve.Owens@disney.com>:
>>>>> One step closer
>>>>>
>>>>> VIO *
>>>>>
>>>>> PluginVC::do_io_write(Continuation * c, int64_t nbytes, IOBufferReader
>>>>>*
>>>>> abuffer, bool owner)
>>>>>
>>>>> {
>>>>>
>>>>>
>>>>>   ink_assert(!closed);
>>>>>
>>>>>   ink_assert(magic == PLUGIN_VC_MAGIC_ALIVE);
>>>>>
>>>>>
>>>>>   if (abuffer) {
>>>>>
>>>>>     ink_assert(!owner);
>>>>>
>>>>>     write_state.vio.buffer.reader_for(abuffer);
>>>>>
>>>>>   } else {
>>>>>
>>>>>     write_state.vio.buffer.clear();
>>>>>
>>>>>   }
>>>>>
>>>>>
>>>>>   // Note: we set vio.op last because process_write_side looks at it
to
>>>>>
>>>>>   //  tell if the VConnection is active.
>>>>>
>>>>>   write_state.vio.mutex = c->mutex;
>>>>>
>>>>>   write_state.vio._cont = c;
>>>>>
>>>>>   write_state.vio.nbytes = nbytes;
>>>>>
>>>>>   write_state.vio.ndone = 0;
>>>>>
>>>>>   write_state.vio.vc_server = (VConnection *) this;
>>>>>
>>>>>   write_state.vio.op = VIO::WRITE;
>>>>>
>>>>>
>>>>>   Debug("pvc", "[%u] %s: do_io_write for %"PRId64" bytes", PVC_ID,
>>>>>PVC_TYPE,
>>>>> nbytes);
>>>>>
>>>>>
>>>>>   // Since reentrant callbacks are not allowed on from do_io
>>>>>
>>>>>   //   functions schedule ourselves get on a different stack
>>>>>
>>>>>   need_write_process = true;
>>>>>
>>>>>   setup_event_cb(0, &sm_lock_retry_event);
>>>>>
>>>>>
>>>>>   return &write_state.vio;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> It would appear that the value passed in to the third argument of
>>>>> TSVConnWrite ends up being assigned to write_state.vio.nbytes
>>>>>
>>>>> the docs on write_state.vio.nbytes state:
>>>>>
>>>>> /**
>>>>>
>>>>>     Number of bytes to be done for this operation.
>>>>>
>>>>>
>>>>>     The total number of bytes this operation must complete.
>>>>>
>>>>>
>>>>>   */
>>>>>
>>>>>   int64_t nbytes;
>>>>>
>>>>>
>>>>> So far clear as mud.  Apologies in advance for my mental density.
>>>>>
>>>>>
>>>>>
>>>>> From: "Owens, Steve" <Steve.Owens@disney.com>
>>>>> Reply-To: "users@trafficserver.apache.org"
>>>>><users@trafficserver.apache.org>
>>>>> Date: Fri, 25 Jan 2013 11:09:08 -0800
>>>>> To: "users@trafficserver.apache.org" <users@trafficserver.apache.org>
>>>>> Subject: Re: Api Questions
>>>>>
>>>>> An addendum to the previous message I have resorted to reading the
>>>>>traffic
>>>>> server code in order to try to discern what it does:
>>>>>
>>>>> I am examining
>>>>>
>>>>> TSVIO
>>>>>
>>>>> TSVConnWrite(TSVConn connp, TSCont contp, TSIOBufferReader readerp,
>>>>>int64_t
>>>>> nbytes)
>>>>>
>>>>> {
>>>>>
>>>>>   sdk_assert(sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS);
>>>>>
>>>>>   sdk_assert(sdk_sanity_check_iocore_structure(contp) == TS_SUCCESS);
>>>>>
>>>>>   sdk_assert(sdk_sanity_check_iocore_structure(readerp) == TS_SUCCESS);
>>>>>
>>>>>   sdk_assert(nbytes >= 0);
>>>>>
>>>>>
>>>>>   FORCE_PLUGIN_MUTEX(contp);
>>>>>
>>>>>   VConnection *vc = (VConnection *) connp;
>>>>>
>>>>>
>>>>>   return reinterpret_cast<TSVIO>(vc->do_io_write((INKContInternal
*)
>>>>>contp,
>>>>> nbytes, (IOBufferReader *) readerp));
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> It would appear that the answer to what this method does lies in what
>>>>> happens in the function "vc->do_io_write" yet when I search the
>>>>> trafficserver 3.2.0 codebase for any file that actually contains an
>>>>> implementation of "vc->do_io_write" I find that "do_io_write" is
>>>>>declared in
>>>>> several .h files but not defined in any .c or .cpp files.
>>>>>
>>>>>
>>>>> Does anyone know where in the code base the implementation of
>>>>>"do_io_write"
>>>>> resides?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: "Owens, Steve" <Steve.Owens@disney.com>
>>>>> Reply-To: "users@trafficserver.apache.org"
>>>>><users@trafficserver.apache.org>
>>>>> Date: Fri, 25 Jan 2013 10:48:11 -0800
>>>>> To: "users@trafficserver.apache.org" <users@trafficserver.apache.org>
>>>>> Subject: Api Questions
>>>>>
>>>>> I am trying to understand the relationship between the following
>>>>>functions
>>>>> as they are used within a transformation plugin and so far no matter
>>>>>how
>>>>> hard I dig the mystery remains.  I am hoping that people who have a
>>>>>better
>>>>> understanding of ATS than I do would be willing to help clear up any
>>>>> misunderstandings I have.
>>>>>
>>>>> Here are the functions I find confusing:
>>>>>
>>>>> TSVIONBytesSet
>>>>>
>>>>> (TSIOBufferCopy and or TSIOBufferWrite)
>>>>>
>>>>> And finally
>>>>>
>>>>> TSVConnWrite
>>>>>
>>>>>
>>>>> Each of the above functions takes an int_64 argument at the end to
>>>>>indicate
>>>>> the number of bytes but I don't fully understand the purpose of this
>>>>> parameter with respect to each of the functions.
>>>>>
>>>>>
>>>>> Suppose I have a transformation which is intended to write a prefix
>>>>>spew the
>>>>> response and then write a suffix to the down stream I would imagine it
>>>>>looks
>>>>> something like this:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> void handleTransformOperation(TSCont contp) {
>>>>>
>>>>> TSIOBuffer buf_test;
>>>>>
>>>>> int_64 written = 0;
>>>>>
>>>>> TransormationData* pData = (TransformationData*) TSContDataGet(contp);
>>>>>
>>>>> TSVIO input_vio = TSTransformOutputVConnGet(contp);
>>>>>
>>>>>
>>>>> if(data->state == STATE_BEGIN) {
>>>>>
>>>>>     // Initialize the output_buffer and output_reader
>>>>>
>>>>>     data->output_buffer = TSIOBufferCreate();
>>>>>
>>>>>     data->output_reader = TSIOBufferReaderAlloc(data->output_buffer);
>>>>>
>>>>>
>>>>>
>>>>>    // Write out a prefix
>>>>>
>>>>>    written += TSIOBufferWrite(data->output_buffer, "{Hello I am a
>>>>>prefix}",
>>>>> strlen(prefix) * sizeof(char));
>>>>>
>>>>>    data->state == STATE_MIDDLE;
>>>>>
>>>>>
>>>>>    // Figure out if the upstream wants to stop receiving WRITEREADY and
>>>>> WRITE_COMPLETE events
>>>>>
>>>>>    buf_test = TSVIOBufferGet(input_vio);
>>>>>
>>>>>    if (!buf_test) {
>>>>>
>>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>>
>>>>>        TSVIONBytesSet(data->output_vio, /*What goes here doneget or
>>>>>written
>>>>> + doneget or just written?  Bonus question: why? /* ???);
>>>>>
>>>>>        data->state = STATE_END;
>>>>>
>>>>>    } else
>>>>>
>>>>>        data->state = STATE_MIDDLE;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> // Notice that the above if falls through to the next state so it is
>>>>> possible that the following code may or may not
>>>>>
>>>>> // get executed based on the buf_test result above.
>>>>>
>>>>> if(data->state == STATE_MIDDLE) {
>>>>>
>>>>>    // Figure out if the upstream wants to stop receiving WRITEREADY and
>>>>> WRITE_COMPLETE events
>>>>>
>>>>>    buf_test = TSVIOBufferGet(input_vio);
>>>>>
>>>>>    if (!buf_test) {
>>>>>
>>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>>
>>>>>        TSVIONBytesSet(data->output_vio, /*What goes here doneget or
>>>>>written
>>>>> + doneget or just written?  Bonus question: why? /* ???);
>>>>>
>>>>>        data->state = STATE_END;
>>>>>
>>>>>    } else {
>>>>>
>>>>>        // Figure out how much data is left to read from the upstream
>>>>>
>>>>>        int64_t upstreamBytesRemaining = TSVIONTodoGet(input_vio);
>>>>>
>>>>>        int64_t upstreamBytesAvail =
>>>>> TSIOBufferReaderAvail(TSVIOReaderGet(input_vio));
>>>>>
>>>>>        if(upstreamBytesRemaining > 0) {
>>>>>
>>>>>             int64_t bytesToWrite = upstreamBytesRemaining;
>>>>>
>>>>>             if(bytesToWrite > upstreamBytesAvail)
>>>>>
>>>>>                  bytesToWrite = upstreamBytesAvail;
>>>>>
>>>>>
>>>>>
>>>>>             /* Copy the data from the read buffer to the output buffer.
>>>>>*/
>>>>>
>>>>>     TSIOBufferCopy(TSVIOBufferGet(data->output_vio),
>>>>> TSVIOReaderGet(input_vio), bytesToWrite, 0);
>>>>>
>>>>>
>>>>>            /* Tell the read buffer that we have read the data and are
>>>>>no
>>>>>
>>>>>     * longer interested in it.
>>>>>
>>>>>     */
>>>>>
>>>>>     TSIOBufferReaderConsume(TSVIOReaderGet(input_vio), bytesToWrite);
>>>>>
>>>>>
>>>>>            /* Modify the input VIO to reflect how much data we've
>>>>>
>>>>>     * completed.
>>>>>
>>>>>     */
>>>>>
>>>>>            int64_t doneget = TSVIONDoneGet(input_vio);
>>>>>
>>>>>    TSVIONDoneSet(input_vio, /* This is where things get really muddy
>>>>>what
>>>>> goes here?  doneget or doneget + bytesToWrite or doneget + bytesToWrite
>>>>>+
>>>>> written */ ??);
>>>>>
>>>>>            /* What is the impact of calling this later in the handler
>>>>>if it
>>>>> has already been called here?
>>>>>
>>>>>        }
>>>>>
>>>>>
>>>>>        /* Now we check the input VIO to see if there is data left to
>>>>>
>>>>> * read.
>>>>>
>>>>> */
>>>>>
>>>>>         upstreamBytesRemaining = TSVIONTodoGet(input_vio);
>>>>>
>>>>> if (upstreamBytesRemaining > 0) {
>>>>>
>>>>>     if (upstreamBytesAvail > 0) {
>>>>>
>>>>>       /* If there is data left to read, then we reenable the output
>>>>>
>>>>>        * connection by reenabling the output VIO. This will wake up
>>>>>
>>>>>        * the output connection and allow it to consume data from the
>>>>>
>>>>>        * output buffer.
>>>>>
>>>>>        */
>>>>>
>>>>>        TSVIOReenable(data->output_vio);
>>>>>
>>>>>
>>>>>        /* Call back the input VIO continuation to let it know that we
>>>>>
>>>>>         * are ready for more data.
>>>>>
>>>>>         */
>>>>>
>>>>>        TSContCall(TSVIOContGet(input_vio),
>>>>>
>>>>> TS_EVENT_VCONN_WRITE_READY, input_vio);
>>>>>
>>>>>     }
>>>>>
>>>>> } else {
>>>>>
>>>>>     /* There is no data left to read so set state to STATE_END and */
>>>>>
>>>>>     data->state = STATE_END;
>>>>>
>>>>> }
>>>>>
>>>>>    }
>>>>>
>>>>>
>>>>>    // The above can fall through to this if block
>>>>>
>>>>>    if(data->state == STATE_END) {
>>>>>
>>>>>        /* Spew a suffix */
>>>>>
>>>>>        const char* suffix = "{i am a suffix}";
>>>>>
>>>>>        int64_t localWritten = TSIOBufferWrite(data->output_buffer,
>>>>>suffix,
>>>>> (strlen(suffix) - 1) * sizeof(char));
>>>>>
>>>>>
>>>>>
>>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>>
>>>>>        TSVIONBytesSet(data->output_vio, /* What goes here? Why? */??);
>>>>>
>>>>>        TSVIOReenable(data->output_vio);
>>>>>
>>>>>
>>>>>       /* Call back the input VIO continuation to let it know that we
>>>>>
>>>>>        * have completed the write operation.
>>>>>
>>>>>        */
>>>>>
>>>>>        TSContCall(TSVIOContGet(input_vio),
>>>>>TS_EVENT_VCONN_WRITE_COMPLETE,
>>>>> input_vio);
>>>>>
>>>>>    }
>>>>>
>>>>>
>>>>>    /* I am presuming we only want to call this once per invocation of
>>>>> handlerOperation  But I am not sure */
>>>>>
>>>>>   data->output_vio = TSVConnWrite(output_conn, contp,
>>>>>data->output_reader,
>>>>> /* What to put here hmmm and why? */ ?? );
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> I really want to understand how these methods relate and what they do.
>>>>>I
>>>>> would be willing to help update/contribute to the ATS docs but first
I
>>>>>would
>>>>> need to understand what they do and so far the examples that are
>>>>>available
>>>>> have not made it clear to me what is going on as yet so any help would
>>>>>be
>>>>> appreciated.
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>

Mime
View raw message