jmeter-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Pokhilko <a...@ya.ru>
Subject Re: JMeter DSL discussion
Date Wed, 10 Aug 2016 18:37:37 GMT
Jeremiah,

Thanks for so detailed explanation.

As a side note, did you see Taurus project (http://gettaurus.org/),
which tries to bring some YAML-based way to express JMeter test plans?

Andrey Pokhilko

On 08/10/2016 08:49 PM, Epp, Jeremiah W (Contractor) wrote:
>> -----Original Message-----
>> From: Philippe Mouawad [mailto:philippe.mouawad@gmail.com]
>> Sent: Thursday, July 28, 2016 4:37 PM
>> To: dev@jmeter.apache.org
>> Subject: Re: JMeter DSL discussion
>>
>> I think DSL has 2 important advantages:
>> - It is more readable in source repositories.
>> - it is better for developers who load test their application in CI/CD.
>> This is clearly an important move JMeter must follow.
> If I might chime in on this conversation?   I'd like to offer the
> perspective of an organisation actually using JMeter with CI right now.
>
> To preface, we're doing something rather unusual in that we're actually
> generating test plans in CI from our functional tests using a command-line
> recorder daemon I wrote (based on JMeter; I'm still trying to get permission
> to release the source) and then correlating them with post-processing
> scripts.
>
> In this capacity, we would be VERY interested in the ability to record to
> and operate on a format that is not JMX.  From our perspective, a robust and
> well-considered test plan DSL brings two major advantages to the table:
> determinism and structural soundness.
>
> By determinism, I mean that every node of the test plan tree is consistently
> represented.  By structural soundness, I mean that every relationship
> between nodes arises as a result of the structure expressed in the test plan
> file itself (i.e. implementation details don't create implicit structure).
>
> To elaborate further, currently, test plans are written to JMX files.  JMX
> is an XML-based format which, if you look at its actual structure, is
> moderately insane, horrendously verbose, and basically undocumented.  After
> several months of working with it, I believe it's better to consider it a
> serialisation of a Java data structure that happens to be in text form more
> than an actual file format.  I would personally love nothing more than to
> see it deprecated.
>
> If you haven't looked recently, I'll recap: most testElements use stringProp
> and friends with a name property to configure their nodes, and the
> parent/child relationship between most elements is implicit, relying on the
> hashTree immediately following it in the file.  The actual tree-like nature
> of an XML document is largely ignored in many cases within the format.
> (Also, again, it's XML, so there are invalid characters, which is bad.)
>
> For example, this is the structure of a single HTTP request with one
> Argument (and some of my own comments):
>
>         <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy"
testname="9952 /as/authorization.oauth2" enabled="true">
>           <elementProp  name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel"
testclass="Arguments" enabled="true"> 
>             <collectionProp name="Arguments.arguments">
>               <elementProp name="state" elementType="HTTPArgument"> <!-- Note
the redundancies, particularly the names -->
>                 <boolProp name="HTTPArgument.always_encode">false</boolProp>
>                 <stringProp name="Argument.name">state</stringProp>
>                 <stringProp name="Argument.value">${state009951}</stringProp>
>                 <stringProp name="Argument.metadata">=</stringProp>
>                 <boolProp name="HTTPArgument.use_equals">true</boolProp>
>               </elementProp>
>             </collectionProp>
>           </elementProp>
>           <stringProp /> <!-- I'll omit all the attributes and values from here
onward; I think you get the idea -->
> <!-- Nine more stringProp here -->
>           <boolProp />
> <!-- Four more boolProp here -->
>         </HTTPSamplerProxy>
>         <hashTree>
>           <HeaderManager>
>             <collectionProp>
>               <elementProp>
>                 <stringProp />
>                 <stringProp />
>               </elementProp>
>              <!-- Four more elementProp with two stringProp each here -->
>             </collectionProp>
>           </HeaderManager>
>           <hashTree/> <!-- This empty hashTree is NOT optional -->
>           <RegexExtractor>
>             <stringProp />
> <!-- Six more stringProp here -->
>           </RegexExtractor>
>           <hashTree/> <!-- This empty hashTree at the end IS optional -->
>         </hashTree>
>
> Unaltered, this single request description is 107 lines and 7208 bytes.
>
> Note the <hashTree> after the </HTTPSamplerProxy> (and many -- but not all!
> -- other elements) is understood by JMeter as containing the _children_ of
> the <HTTPSamplerProxy>.  For elements with no children, it's still necessary
> that they be followed by an empty <hashTree/> if there are more elements at
> the same level of the tree.  (This is, I believe, a consequence of a
> TestPlan's internal representation as a ListedHashTree of ListedHashTrees.)
> As far as we've been able to determine, this format is entirely
> undocumented.
>
> The consequence of all this is it is EXTREMELY difficult to reliably operate
> on a JMX file with external tools, and much of it has been trial-and-error.
> Really, it's been a nightmare.  A DSL should have clean and clear semantics
> that prevent parsing ambiguity and speed development time.  Anything that
> does not is a wasted effort.
>
> So bearing all that in mind, I would strongly recommend this additional
> requirement: a new DSL MUST decouple the test plan format from
> implementation details.  A good example of how this is bad in JMX is all the
> attribute noise on any sampler-- the test plan shouldn't have to know about
> the guiclass, testclass, elementType, etc.  It's all obvious from the
> structure and the type of the node.
>
>> - the language it should be developed in. My personal preference would go
>> for Groovy but I am not an expert in DSLs.
> Hang on, why would you develop your DSL in a different language from what
> JMeter itself is actually written in?  This wold seem to  add an entire new
> toolchain to the build process.
>
>> - how to express what today we can do through JSR223 languages which might
>> be the most complex think to translate
> Is this really an issue?  This seems orthogonal to the goal of creating a
> language to describe _test plans_ themselves.  As long as this DSL has a way
> to express the JSR223 sampler, saving the actual code that goes in it should
> go without saying.
>
> Regards,
> Wyatt
>
> Confidentiality Notice: This electronic message transmission, including any attachment(s),
may contain confidential, proprietary, or privileged information from Chemical Abstracts Service
("CAS"), a division of the American Chemical Society ("ACS"). If you have received this transmission
in error, be advised that any disclosure, copying, distribution, or use of the contents of
this information is strictly prohibited. Please destroy all copies of the message and contact
the sender immediately by either replying to this message or calling 614-447-3600.
>


Mime
View raw message