trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owens, Steve" <Steve.Ow...@disney.com>
Subject Re: Api Questions
Date Fri, 25 Jan 2013 20:44:55 GMT
Thank you Otto I appreciate the help you have given.

To the wider audience should any one be interested:

One of the things that would be really helpful is if someone who really
knew the code base would take the time to actually provide meaningful
comments on the API functions.

For example from the file InkIOCoreAPI.cc we have

/* VIOs */

void
TSVIOReenable(TSVIO viop)
{
  sdk_assert(sdk_sanity_check_iocore_structure(viop) == TS_SUCCESS);

  VIO *vio = (VIO *)viop;
  vio->reenable();
}


It seems to me that if this is an API function intended to be used by a
wide audience that it ought to have some quality comments that explains
what the function does?

In fact if you examine the entire file InkIOcoreAPI.cc (Traffic Sever
version 3.2.0) you will find that not a single function in this file is
documented yet these are some of the core API functions needed to write
Plugin code.  I would be happy to contribute by commenting the code at
some point but at this point I don't feel qualified to write accurate
comments.

The header file in ts.h doesn't add much to the clarity.  Starting at line
2721 we have:
  /*
--------------------------------------------------------------------------
     VIOs */
  tsapi void TSVIOReenable(TSVIO viop);
  tsapi TSIOBuffer TSVIOBufferGet(TSVIO viop);
  tsapi TSIOBufferReader TSVIOReaderGet(TSVIO viop);
  tsapi int64_t TSVIONBytesGet(TSVIO viop);
  tsapi void TSVIONBytesSet(TSVIO viop, int64_t nbytes);
  tsapi int64_t TSVIONDoneGet(TSVIO viop);
  tsapi void TSVIONDoneSet(TSVIO viop, int64_t ndone);
  tsapi int64_t TSVIONTodoGet(TSVIO viop);
  tsapi TSMutex TSVIOMutexGet(TSVIO viop);
  tsapi TSCont TSVIOContGet(TSVIO viop);
  tsapi TSVConn TSVIOVConnGet(TSVIO viop);

  /*
--------------------------------------------------------------------------



That is not a lot of comments in my book.



On 1/25/13 12:34 PM, "Otto van der Schaaf" <oschaaf@gmail.com> wrote:

>I hope that works out, might be helpful and prevent other from having
>to go down the same road :-) I would be happy to help out with that.
>
>Regarding INKVIONBytesGet:
>
>"Gets the number of bytes to be performed by the IO operation
>described by viop. This is the nbytes parameter that's passed to
>INKVConnRead or INKVConnWrite."
>
>So, no, I don't think that modifying anything. If I remember
>correctly, that call returns the total number of bytes that the will
>be written to the passed in vio. So in the context of the picture
>above, you will write as much bytes to the output as you get from the
>input I guess. You can specify INT64_MAX if you don't know yet how
>much bytes you will write, in which case trafficserver will either
>send a connection:close header or send a transfer-encoding chunked.
>
>You might want to have a look at the gzip plugin in /experimental: I
>made an effort to make to code a little more understandable then the
>null-transform, and I think it's also a little more experimental :-)
>
>
>Regards,
>
>Otto
>
>
>2013/1/25 Owens, Steve <Steve.Owens@disney.com>:
>> Otto,
>>
>> I have started mapping code examples from the null transform example to
>> the diagram to try to connect the dots see attached.
>>
>> Steve Owens
>>
>>
>>
>> On 1/25/13 11:44 AM, "Owens, Steve" <Steve.Owens@disney.com> wrote:
>>
>>>Otto, yes I have seen them and read them several times.  The second link
>>>explains in rather abstract terms what VIO's do and how they relate to
>>>buffers.
>>>
>>>But I personally could use a little more amplification on what the
>>>specific methods are that manipulate these structures and what effect
>>>that
>>>calls to these methods have on the structures depicted in the images.
>>>
>>>The null transform plugin re-uses variables and takes enough shortcuts
>>>that what it is doing is seems obscured and I am trying to pierce that
>>>obscurity.  I will continue to try to pierce the veil but after having
>>>done so I would love to write up additional info to make it clearer to
>>>dunderheads such as myself what these methods actually do in order to
>>>reduce the slope of the learning curve for this technology.
>>>
>>>Thank you for re-directing me to these I am sure that given enough time
>>>and patience I will crack the code.
>>>
>>>
>>>
>>>On 1/25/13 11:25 AM, "Otto van der Schaaf" <oschaaf@gmail.com> wrote:
>>>
>>>>Hi Steve,
>>>>
>>>>Did you get a chance already to have a look at
>>>>http://trafficserver.apache.org/docs/v2/sdk/images/vconnection.jpg ?
>>>>That image is from
>>>>http://trafficserver.apache.org/docs/v2/sdk/HTTPTransformationPlugins.h
>>>>tm
>>>>l
>>>>#VIOs
>>>>Both helped my understanding of the transformation api's
>>>>
>>>>Regards,
>>>>
>>>>Otto
>>>>
>>>>
>>>>2013/1/25 Owens, Steve <Steve.Owens@disney.com>:
>>>>> One step closer
>>>>>
>>>>> VIO *
>>>>>
>>>>> PluginVC::do_io_write(Continuation * c, int64_t nbytes,
>>>>>IOBufferReader
>>>>>*
>>>>> abuffer, bool owner)
>>>>>
>>>>> {
>>>>>
>>>>>
>>>>>   ink_assert(!closed);
>>>>>
>>>>>   ink_assert(magic == PLUGIN_VC_MAGIC_ALIVE);
>>>>>
>>>>>
>>>>>   if (abuffer) {
>>>>>
>>>>>     ink_assert(!owner);
>>>>>
>>>>>     write_state.vio.buffer.reader_for(abuffer);
>>>>>
>>>>>   } else {
>>>>>
>>>>>     write_state.vio.buffer.clear();
>>>>>
>>>>>   }
>>>>>
>>>>>
>>>>>   // Note: we set vio.op last because process_write_side looks at it
>>>>>to
>>>>>
>>>>>   //  tell if the VConnection is active.
>>>>>
>>>>>   write_state.vio.mutex = c->mutex;
>>>>>
>>>>>   write_state.vio._cont = c;
>>>>>
>>>>>   write_state.vio.nbytes = nbytes;
>>>>>
>>>>>   write_state.vio.ndone = 0;
>>>>>
>>>>>   write_state.vio.vc_server = (VConnection *) this;
>>>>>
>>>>>   write_state.vio.op = VIO::WRITE;
>>>>>
>>>>>
>>>>>   Debug("pvc", "[%u] %s: do_io_write for %"PRId64" bytes", PVC_ID,
>>>>>PVC_TYPE,
>>>>> nbytes);
>>>>>
>>>>>
>>>>>   // Since reentrant callbacks are not allowed on from do_io
>>>>>
>>>>>   //   functions schedule ourselves get on a different stack
>>>>>
>>>>>   need_write_process = true;
>>>>>
>>>>>   setup_event_cb(0, &sm_lock_retry_event);
>>>>>
>>>>>
>>>>>   return &write_state.vio;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> It would appear that the value passed in to the third argument of
>>>>> TSVConnWrite ends up being assigned to write_state.vio.nbytes
>>>>>
>>>>> the docs on write_state.vio.nbytes state:
>>>>>
>>>>> /**
>>>>>
>>>>>     Number of bytes to be done for this operation.
>>>>>
>>>>>
>>>>>     The total number of bytes this operation must complete.
>>>>>
>>>>>
>>>>>   */
>>>>>
>>>>>   int64_t nbytes;
>>>>>
>>>>>
>>>>> So far clear as mud.  Apologies in advance for my mental density.
>>>>>
>>>>>
>>>>>
>>>>> From: "Owens, Steve" <Steve.Owens@disney.com>
>>>>> Reply-To: "users@trafficserver.apache.org"
>>>>><users@trafficserver.apache.org>
>>>>> Date: Fri, 25 Jan 2013 11:09:08 -0800
>>>>> To: "users@trafficserver.apache.org" <users@trafficserver.apache.org>
>>>>> Subject: Re: Api Questions
>>>>>
>>>>> An addendum to the previous message I have resorted to reading the
>>>>>traffic
>>>>> server code in order to try to discern what it does:
>>>>>
>>>>> I am examining
>>>>>
>>>>> TSVIO
>>>>>
>>>>> TSVConnWrite(TSVConn connp, TSCont contp, TSIOBufferReader readerp,
>>>>>int64_t
>>>>> nbytes)
>>>>>
>>>>> {
>>>>>
>>>>>   sdk_assert(sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS);
>>>>>
>>>>>   sdk_assert(sdk_sanity_check_iocore_structure(contp) == TS_SUCCESS);
>>>>>
>>>>>   sdk_assert(sdk_sanity_check_iocore_structure(readerp) ==
>>>>>TS_SUCCESS);
>>>>>
>>>>>   sdk_assert(nbytes >= 0);
>>>>>
>>>>>
>>>>>   FORCE_PLUGIN_MUTEX(contp);
>>>>>
>>>>>   VConnection *vc = (VConnection *) connp;
>>>>>
>>>>>
>>>>>   return reinterpret_cast<TSVIO>(vc->do_io_write((INKContInternal
*)
>>>>>contp,
>>>>> nbytes, (IOBufferReader *) readerp));
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> It would appear that the answer to what this method does lies in what
>>>>> happens in the function "vc->do_io_write" yet when I search the
>>>>> trafficserver 3.2.0 codebase for any file that actually contains an
>>>>> implementation of "vc->do_io_write" I find that "do_io_write" is
>>>>>declared in
>>>>> several .h files but not defined in any .c or .cpp files.
>>>>>
>>>>>
>>>>> Does anyone know where in the code base the implementation of
>>>>>"do_io_write"
>>>>> resides?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: "Owens, Steve" <Steve.Owens@disney.com>
>>>>> Reply-To: "users@trafficserver.apache.org"
>>>>><users@trafficserver.apache.org>
>>>>> Date: Fri, 25 Jan 2013 10:48:11 -0800
>>>>> To: "users@trafficserver.apache.org" <users@trafficserver.apache.org>
>>>>> Subject: Api Questions
>>>>>
>>>>> I am trying to understand the relationship between the following
>>>>>functions
>>>>> as they are used within a transformation plugin and so far no matter
>>>>>how
>>>>> hard I dig the mystery remains.  I am hoping that people who have a
>>>>>better
>>>>> understanding of ATS than I do would be willing to help clear up any
>>>>> misunderstandings I have.
>>>>>
>>>>> Here are the functions I find confusing:
>>>>>
>>>>> TSVIONBytesSet
>>>>>
>>>>> (TSIOBufferCopy and or TSIOBufferWrite)
>>>>>
>>>>> And finally
>>>>>
>>>>> TSVConnWrite
>>>>>
>>>>>
>>>>> Each of the above functions takes an int_64 argument at the end to
>>>>>indicate
>>>>> the number of bytes but I don't fully understand the purpose of this
>>>>> parameter with respect to each of the functions.
>>>>>
>>>>>
>>>>> Suppose I have a transformation which is intended to write a prefix
>>>>>spew the
>>>>> response and then write a suffix to the down stream I would imagine
>>>>>it
>>>>>looks
>>>>> something like this:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> void handleTransformOperation(TSCont contp) {
>>>>>
>>>>> TSIOBuffer buf_test;
>>>>>
>>>>> int_64 written = 0;
>>>>>
>>>>> TransormationData* pData = (TransformationData*)
>>>>>TSContDataGet(contp);
>>>>>
>>>>> TSVIO input_vio = TSTransformOutputVConnGet(contp);
>>>>>
>>>>>
>>>>> if(data->state == STATE_BEGIN) {
>>>>>
>>>>>     // Initialize the output_buffer and output_reader
>>>>>
>>>>>     data->output_buffer = TSIOBufferCreate();
>>>>>
>>>>>     data->output_reader = TSIOBufferReaderAlloc(data->output_buffer);
>>>>>
>>>>>
>>>>>
>>>>>    // Write out a prefix
>>>>>
>>>>>    written += TSIOBufferWrite(data->output_buffer, "{Hello I am a
>>>>>prefix}",
>>>>> strlen(prefix) * sizeof(char));
>>>>>
>>>>>    data->state == STATE_MIDDLE;
>>>>>
>>>>>
>>>>>    // Figure out if the upstream wants to stop receiving WRITEREADY
>>>>>and
>>>>> WRITE_COMPLETE events
>>>>>
>>>>>    buf_test = TSVIOBufferGet(input_vio);
>>>>>
>>>>>    if (!buf_test) {
>>>>>
>>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>>
>>>>>        TSVIONBytesSet(data->output_vio, /*What goes here doneget or
>>>>>written
>>>>> + doneget or just written?  Bonus question: why? /* ???);
>>>>>
>>>>>        data->state = STATE_END;
>>>>>
>>>>>    } else
>>>>>
>>>>>        data->state = STATE_MIDDLE;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> // Notice that the above if falls through to the next state so it is
>>>>> possible that the following code may or may not
>>>>>
>>>>> // get executed based on the buf_test result above.
>>>>>
>>>>> if(data->state == STATE_MIDDLE) {
>>>>>
>>>>>    // Figure out if the upstream wants to stop receiving WRITEREADY
>>>>>and
>>>>> WRITE_COMPLETE events
>>>>>
>>>>>    buf_test = TSVIOBufferGet(input_vio);
>>>>>
>>>>>    if (!buf_test) {
>>>>>
>>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>>
>>>>>        TSVIONBytesSet(data->output_vio, /*What goes here doneget or
>>>>>written
>>>>> + doneget or just written?  Bonus question: why? /* ???);
>>>>>
>>>>>        data->state = STATE_END;
>>>>>
>>>>>    } else {
>>>>>
>>>>>        // Figure out how much data is left to read from the upstream
>>>>>
>>>>>        int64_t upstreamBytesRemaining = TSVIONTodoGet(input_vio);
>>>>>
>>>>>        int64_t upstreamBytesAvail =
>>>>> TSIOBufferReaderAvail(TSVIOReaderGet(input_vio));
>>>>>
>>>>>        if(upstreamBytesRemaining > 0) {
>>>>>
>>>>>             int64_t bytesToWrite = upstreamBytesRemaining;
>>>>>
>>>>>             if(bytesToWrite > upstreamBytesAvail)
>>>>>
>>>>>                  bytesToWrite = upstreamBytesAvail;
>>>>>
>>>>>
>>>>>
>>>>>             /* Copy the data from the read buffer to the output
>>>>>buffer.
>>>>>*/
>>>>>
>>>>>     TSIOBufferCopy(TSVIOBufferGet(data->output_vio),
>>>>> TSVIOReaderGet(input_vio), bytesToWrite, 0);
>>>>>
>>>>>
>>>>>            /* Tell the read buffer that we have read the data and are
>>>>>no
>>>>>
>>>>>     * longer interested in it.
>>>>>
>>>>>     */
>>>>>
>>>>>     TSIOBufferReaderConsume(TSVIOReaderGet(input_vio), bytesToWrite);
>>>>>
>>>>>
>>>>>            /* Modify the input VIO to reflect how much data we've
>>>>>
>>>>>     * completed.
>>>>>
>>>>>     */
>>>>>
>>>>>            int64_t doneget = TSVIONDoneGet(input_vio);
>>>>>
>>>>>    TSVIONDoneSet(input_vio, /* This is where things get really muddy
>>>>>what
>>>>> goes here?  doneget or doneget + bytesToWrite or doneget +
>>>>>bytesToWrite
>>>>>+
>>>>> written */ ??);
>>>>>
>>>>>            /* What is the impact of calling this later in the handler
>>>>>if it
>>>>> has already been called here?
>>>>>
>>>>>        }
>>>>>
>>>>>
>>>>>        /* Now we check the input VIO to see if there is data left to
>>>>>
>>>>> * read.
>>>>>
>>>>> */
>>>>>
>>>>>         upstreamBytesRemaining = TSVIONTodoGet(input_vio);
>>>>>
>>>>> if (upstreamBytesRemaining > 0) {
>>>>>
>>>>>     if (upstreamBytesAvail > 0) {
>>>>>
>>>>>       /* If there is data left to read, then we reenable the output
>>>>>
>>>>>        * connection by reenabling the output VIO. This will wake up
>>>>>
>>>>>        * the output connection and allow it to consume data from the
>>>>>
>>>>>        * output buffer.
>>>>>
>>>>>        */
>>>>>
>>>>>        TSVIOReenable(data->output_vio);
>>>>>
>>>>>
>>>>>        /* Call back the input VIO continuation to let it know that we
>>>>>
>>>>>         * are ready for more data.
>>>>>
>>>>>         */
>>>>>
>>>>>        TSContCall(TSVIOContGet(input_vio),
>>>>>
>>>>> TS_EVENT_VCONN_WRITE_READY, input_vio);
>>>>>
>>>>>     }
>>>>>
>>>>> } else {
>>>>>
>>>>>     /* There is no data left to read so set state to STATE_END and */
>>>>>
>>>>>     data->state = STATE_END;
>>>>>
>>>>> }
>>>>>
>>>>>    }
>>>>>
>>>>>
>>>>>    // The above can fall through to this if block
>>>>>
>>>>>    if(data->state == STATE_END) {
>>>>>
>>>>>        /* Spew a suffix */
>>>>>
>>>>>        const char* suffix = "{i am a suffix}";
>>>>>
>>>>>        int64_t localWritten = TSIOBufferWrite(data->output_buffer,
>>>>>suffix,
>>>>> (strlen(suffix) - 1) * sizeof(char));
>>>>>
>>>>>
>>>>>
>>>>>        int64_t doneget = TSVIONDoneGet(input_vio);
>>>>>
>>>>>        TSVIONBytesSet(data->output_vio, /* What goes here? Why?
>>>>>*/??);
>>>>>
>>>>>        TSVIOReenable(data->output_vio);
>>>>>
>>>>>
>>>>>       /* Call back the input VIO continuation to let it know that we
>>>>>
>>>>>        * have completed the write operation.
>>>>>
>>>>>        */
>>>>>
>>>>>        TSContCall(TSVIOContGet(input_vio),
>>>>>TS_EVENT_VCONN_WRITE_COMPLETE,
>>>>> input_vio);
>>>>>
>>>>>    }
>>>>>
>>>>>
>>>>>    /* I am presuming we only want to call this once per invocation of
>>>>> handlerOperation  But I am not sure */
>>>>>
>>>>>   data->output_vio = TSVConnWrite(output_conn, contp,
>>>>>data->output_reader,
>>>>> /* What to put here hmmm and why? */ ?? );
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> I really want to understand how these methods relate and what they
>>>>>do.
>>>>>I
>>>>> would be willing to help update/contribute to the ATS docs but first
>>>>>I
>>>>>would
>>>>> need to understand what they do and so far the examples that are
>>>>>available
>>>>> have not made it clear to me what is going on as yet so any help
>>>>>would
>>>>>be
>>>>> appreciated.
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>


Mime
View raw message