plc4x-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Łukasz Dywicki <l...@code-house.org>
Subject Re: AW: AW: AW: Source Time of PLC Value?
Date Wed, 10 Mar 2021 16:14:59 GMT
Hey Chris, hope others still follow this thread!

Whether its fault of tight pockets of industry, decade long production
cycles or something else we can't wait indefinitely for them. Maybe,
just maybe OPC UA will become WSDL/SOAP of industry over time. Slow,
over-bloated, painful in development but in the end common data exchange
layer. (After several cycles of WebService Interoperability/WS-I releases!)

I will try to answer in a bit friday-ish attitude to address the JDBC
argument. The stored procedures and other not-so-great things JDBC
brings into table are in their API because they are in use. Maybe not
always, not by everyone but enough of times to be there. This also keeps
focus of database vendors to stick with it cause it saves them from
doing whole new abstraction layer just to cover own product features.
We are leaving space for custom types in our API/SPI. We also do
optimizations on reading data (S7 batching). We do cover some of JDBC
features even if they are named differently. I could also argue if our
field expressions are less or more complicated than basic SQL selects. ;-)

Regardless of industry or big pharma paying or not we are the ones who
lead the edge. So far mainly serving our small goals. Not the others
cause they are gone and quiet.
Apparently Ben have clear intention to ship better support for OPC UA.
He is paying by his own time to do such. I feel myself obligated to help
him cause I might be another person who will indirectly benefit from his
work.

Best,
Łukasz

On 10.03.2021 15:30, Christofer Dutz wrote:
> Hi Lukaz,
> 
> I would say: If there's a NEED from the industry. They could show how much they need
it, by spending a few bucks on such features I would call enterprise features (I'm not talking
about paying for the full implementation, but at least part of it). And especially, we shouldn't
be guessing for them, what they might be needing. If they want something, they should speak
up and tell us what they really need.
> 
> Personally, I have stopped thinking what the industry might be needing as the industry
never started thinking about what the people behind an open-source project might be needing
(Being able to talk about what they did or paid gigs, for example). I know of several cases
where big companies would love to have something. Like I know that Siemens would love to have
an ADS driver in C ... I also know about several other cases where companies would love to
have certain drivers in certain languages. But staying at the Siemens example: As long as
I have to pay 3000€ for a TIA portal license and there's just no way to provide me with
a free (educational) license or there's not a single € in the budget for a paid gig ...
this will be the last driver version I'll work on. 
> 
> For this reason, I will not continue implementing such enterprise features until I can
see the industry openly admitting that they are using PLC4X or need something and that at
least some of our community members can at least pay some of their bills from their hard work
on this great project (not talking about me here).
> 
> And coming back to your proposal: The way you are proposing to implement this with some
generic stuff ... it would make it difficult to provide the same functionality on other languages.
Right now, the API is so simple that it is easy to port. If you really want to tie me to when
I say: "Sort of like JDBC for PLCs" and because of that want to have the MetaData - then consistently
we should be allowed to run Stored Procedures on PLCs, Have a query language that allows data
joining and aggregation as well as mathematical operations, have cursors for large result
sets, also should we be able to use PLC4X to open terminals on PLCs and execute arbitrary
code on the remote device. Cause this is all stuff I have been doing using JDBC ;-) 
> 
> Chris
>  
> 
> -----Ursprüngliche Nachricht-----
> Von: Łukasz Dywicki <luke@code-house.org> 
> Gesendet: Mittwoch, 10. März 2021 15:05
> An: dev@plc4x.apache.org
> Betreff: Re: AW: AW: Source Time of PLC Value?
> 
> Chris,
> What we discuss is whether there is a need and use for result set metadata or not. Given
initial statement of "being a for of JDBC for industrial devices" and looking at JDBC itself
we should provide it. If we don't do it then we lave our OPC UA users with unanswered trouble
which they need to handle on their own. If we postpone it until implementation need arise
in non-opc drivers (let say Ethernet/IP,
> AB-ETH/DeltaV) we leave our client APIs half baked for anyone who do use even standard
UA capabilities.
> 
> I do agree that simplicity is a key, however what you compare are two completely different
things. MQTT is an thin overlay over TCP transport without any real application layer (it
has even transport in the name).
> OPC UA does both with much more diverse application layer causing much bigger overhead
for both ends of communications. Both MQTT and OPC UA bring completely different approach
and create problems in distinct places.
> 
> Neither I nor anybody else want to sacrifice time on pull request which will not be accepted.
> 
> Kind regards,
> Łukasz
> 
> 
> On 10.03.2021 14:34, Christofer Dutz wrote:
>> Yes,
>>
>> OPC-UA has all of this ... and it's probably one of the reasons so many implementations
differ, and you have to again interpret results. It's also probably a reason why OPC-UA sucks
that badly performance-wise.
>>
>> Just saying that we should be careful what we choose as reference. In contrast typical
MQTT doesn't have all of this and it seems to be gaining more and more traction, just because
it doesn't have all of this overhead.
>>
>> Just my thoughts on this.
>>
>> Chris
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Łukasz Dywicki <luke@code-house.org>
>> Gesendet: Mittwoch, 10. März 2021 12:58
>> An: dev@plc4x.apache.org
>> Betreff: Re: AW: Source Time of PLC Value?
>>
>> Matthias,
>> I do believe there are several points which are relevant and still not possible to
be implemented using available PLC4X API. Metadata might not be most fortunate way, but it
can enable or ease analytical and troubleshooting scenarios.
>>
>> Two major points I see:
>> - Verification of device condition - ie. how does caller know response latency? This
is especially relevant with frequent polling. Currently caller has no measures to know that
and can not measure that in reliable way due to async nature of API.
>> There is a way to implement backpressure technics with callback chains (we can automatically
delay execute call), but first we need to bring measures for that.
>> - Caching of values - when we do cyclic subscriptions or emulation of these through
passive connections. There is no way to make caller code aware of nature of data received.
Unless he code everything on his end.
>> - Per operation timeout settings/automatic retry are another functionalities which
are relevant in case of querying and writing. We do not want every driver to implement it,
so making a metadata and/or decorator would unify this layer between drivers.
>>
>> I speak of above from my own perspective cause I did implement an experimental decorator
API for PLC4X [1] to cope with CANopen arbitration issues I found in devices we interface
with. It allows to catch read/write failures [2] and re-attempt entire operation. It is proven
to work [3] independently of driver. Sadly it has no measures to signal its execution back
to caller. In the end, traceability is pretty tough.
>>
>> Obviously OPC UA has everything sorted out, but device specific protocols often do
not. Even a limited set of metadata can give us a way to start bridging the gap and making
Ben's Milo/OPC UA server much closer to a real thing.
>>
>> Best,
>> Łukasz
>>
>> [1]
>> https://github.com/ConnectorIO/connectorio-addons/tree/master/bundles/
>> org.connectorio.plc4x.decorator/src/main/java/org/connectorio/plc4x/de
>> corator
>> [2]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
>> org.connectorio.plc4x.decorator.retry/src/main/java/org/connectorio/pl
>> c4x/decorator/retry/ReadRetry.java
>> [3]
>> https://github.com/ConnectorIO/connectorio-addons/blob/master/bundles/
>> org.connectorio.plc4x.decorator.retry/src/test/java/org/connectorio/pl
>> c4x/decorator/retry/RetryDecoratorReadTest.java
>>
>>
>> On 08.03.2021 23:15, Matthias Milan Strljic wrote:
>>> Hi all,
>>>
>>> i see there a point of how useful this meta information could be in 
>>> some scenarios. But i would rather go the route of chris. Because I 
>>> see there is more of a risk of an overcomplicated inhomogeneous api 
>>> structure. I would not say that all API should have the same java 
>>> structure on each platform but if you have for all the different 
>>> platforms other featuresets you could also just use partially the 
>>> code generation attempt to allow a separate meta info layer.
>>>
>>> And there would be the question if that is worth the additional feature?
>>> I mean I know we could add especially on the OPC UA side some meta 
>>> information but is there more than the  actual source timestamp? And 
>>> if there is none, would it be not better to use it as the actual 
>>> timestamp of the value?
>>>
>>>
>>> Einen schönen Abend ;)
>>> Greetings Matthias
>>>
>>>
>>>
>>> Am Mo., 8. März 2021 um 14:29 Uhr schrieb Christofer Dutz <
>>> christofer.dutz@c-ware.de>:
>>>
>>>> Hi,
>>>>
>>>> I'm generally more concerned about users expecting us to deliver 
>>>> feature that driver X has for driver Y too.
>>>>
>>>> I won't object, if you think it's worth doing.
>>>>
>>>> Perhaps if you could whip up an example in a feature branch? I think 
>>>> perhaps I was still not understanding what you propose.
>>>>
>>>> Would that be ok?
>>>>
>>>>
>>>> Chris
>>>>
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Lukas Ott <ott.lukas.14@gmail.com>
>>>> Gesendet: Montag, 8. März 2021 14:20
>>>> An: dev@plc4x.apache.org
>>>> Betreff: Re: AW: Source Time of PLC Value?
>>>>
>>>> +1 to Lukasz
>>>>
>>>> Am Mo., 8. März 2021 um 14:09 Uhr schrieb Łukasz Dywicki <
>>>> luke@code-house.org>:
>>>>
>>>>> Hold on for a second.
>>>>>
>>>>> Making options pushed over connection string is a long term recipe 
>>>>> for disaster. Apache Camel is a prime example of what could happen 
>>>>> if you starting with configuration with URIs and dynamic parameters.
>>>>> Most of components distinguish producer (writer) and consumer
>>>>> (receiver) options, most complicated components ended up with 40+ 
>>>>> options settable via URI. Having a mix of PlcValues which are 
>>>>> timestamped and
>>>> 'qualified'
>>>>> and not will complicate encoding logic (how we push above info to 
>>>>> IEC encoder/handler or decorate it?) and how we keep custom types 
>>>>> compatible with above?
>>>>>
>>>>> That is why I rather opt for extending API in a way which does not 
>>>>> force existing drivers to do anything. When any of community 
>>>>> members, regardless if its developer or user, will find it 
>>>>> necessary to be ported in another driver, she or he have right to 
>>>>> do it. It is not only yours duty to keep all drivers and languages the
same.
>>>>>
>>>>> Speaking of which I am also not concerned about making all 
>>>>> languages having the same functionality as languages tend to offer 
>>>>> different options which might be hard to be ported. Lets stick to 
>>>>> basics which are same (connection strings, field definitions, 
>>>>> read/write/subscribe
>>>>> syntax) and let various runtime lead their own ways toward
>>>> implementation.
>>>>>
>>>>> Key point is simple - we won't be able to maintain all languages 
>>>>> equally. We should not hold moving ie. go or java forward because 
>>>>> we can't port new API functionality to C/Python/C#/whenever. If 
>>>>> there is someone who *does* use above in production and *wish* to 
>>>>> have new API parts then that person/organization duty is to do it.
>>>>>
>>>>> Best,
>>>>> Łukasz
>>>>>
>>>>>
>>>>> On 08.03.2021 13:17, Christofer Dutz wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I must admit that I would be in favor of keeping it as simple as
>>>>> possible (max age in the connection string) and to implement more 
>>>>> of the missing parts in plc4x (like the subscription simulation 
>>>>> layer) and hereby getting the drivers we have a bit more aligned, 
>>>>> than to implement more and more special API features, that only one 
>>>>> or two drivers support. Cause this way we're running away form the 
>>>>> project-promise of running a given program at any type of PLC with 
>>>>> any
>>>> protocol.
>>>>>>
>>>>>> I mean ... at the current velocity the project is underway, I 
>>>>>> wouldn't
>>>>> really want to add a new MetaData layer that probably in the end 
>>>>> only
>>>>> 1-2 people will be implementing and which we must not only find out 
>>>>> how to fill with life in the first place, but also port to all of 
>>>>> our supported languages.
>>>>>>
>>>>>> If someone's willing to bring in some serious man/womenpower, I'm

>>>>>> fine
>>>>> with that ... or we add it to some wish-list for future extensions.
>>>>> But I wouldn't want to start something like this right now.
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Łukasz Dywicki <luke@code-house.org>
>>>>>> Gesendet: Montag, 8. März 2021 13:07
>>>>>> An: dev@plc4x.apache.org
>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>
>>>>>> I have not mentioned metadata term in my answer, so credits for 
>>>>>> mining
>>>>> it API go to you. :-)
>>>>>>
>>>>>> Now since you brought it I remember that JDBC has a "result set
>>>>> metadata". That might be thing we still miss. The result code is 
>>>>> most important and available instantly, other information is 
>>>>> supplemental and can be read additionally.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Łukasz
>>>>>>
>>>>>> On 08.03.2021 13:01, Ben Hutcheson wrote:
>>>>>>> +1 for Łukasz‘s metadata approach, it would avoid having to

>>>>>>> +create
>>>>>>> duplicate PLcValue classes and give us a lot of flexibility in

>>>>>>> the
>>>>> future.
>>>>>>>
>>>>>>> On Mon, Mar 8, 2021 at 5:14 AM Christofer Dutz 
>>>>>>> <christofer.dutz@c-ware.de>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> sorry for being late to the party ... KNX is currently consuming

>>>>>>>> all my cycles.
>>>>>>>> Just wanted to add my thoughts to the discussion.
>>>>>>>>
>>>>>>>> Initially I thought about adding this sort of information
to the 
>>>>>>>> API, but then I thought that we have so little protocols

>>>>>>>> supporting this sort of concept that blowing up every PLCValue

>>>>>>>> with this time information just would waste CPU time and
Memory.
>>>>>>>>
>>>>>>>> When it comes to simulating subscriptions by polling in the

>>>>>>>> background, I thought it would be a valid solution to provide
a 
>>>>>>>> default age a value can have in the cache and to make it

>>>>>>>> possible to override this in the connection string options.
If a 
>>>>>>>> value is too old,
>>>>> a return code of:
>>>>>>>> STALE_DATA could be returned.
>>>>>>>>
>>>>>>>> Also, we do have an option of adding an additional set of

>>>>>>>> PlcValue
>>>>> types.
>>>>>>>> If a driver supports this quality-of-service type of data,
we 
>>>>>>>> could return special versions of PlcValues, that have these
>>>> properties?
>>>>>>>>
>>>>>>>> This way we don't have to waste the CPU time and Memory for

>>>>>>>> protocols that don't support this concept (We could even

>>>>>>>> implement the methods on the default PlvValues to simply
return 
>>>>>>>> some constants to the PlcValues all have the same API.
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Łukasz Dywicki <luke@code-house.org>
>>>>>>>> Gesendet: Montag, 8. März 2021 10:43
>>>>>>>> An: dev@plc4x.apache.org
>>>>>>>> Betreff: Re: Source Time of PLC Value?
>>>>>>>>
>>>>>>>> Looking at issue we could utilize marker interfaces.
>>>>>>>> At least in Java we could define Qualified and Timestamped
types.
>>>>>>>>
>>>>>>>> Yet looking at present state of API we are unlikely to make
each 
>>>>>>>> and every PlcValue in each and every variant (qualified and

>>>>>>>> timestamped, timestamped, qualified). It would force quite
a lot 
>>>>>>>> of
>>>> wrapping.
>>>>>>>> Safest option I see is to extend subscription/response API
with 
>>>>>>>> additional method to retrieve all markers. For most of drivers

>>>>>>>> it would return just an empty set. It leaves us also an open

>>>>>>>> door for future drivers (or devices) to ship additional piece
of 
>>>>>>>> metadata which
>>>>> does not fit into present API.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Łukasz
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08.03.2021 04:30, Otto Fowler wrote:
>>>>>>>>> So, maybe the extension is timestamp *and* timestamp
source.
>>>>>>>>>
>>>>>>>>> uint64_t timestamp;
>>>>>>>>> typedef enum {
>>>>>>>>>       /* timestamp was generated by the source device
*/
>>>>>>>>>       SOURCE,
>>>>>>>>>       /* timestamp was generated by plc4x on receiving
*/
>>>>>>>>>       GENERATED,
>>>>>>>>> } TimestampQuality;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Mar 5, 2021, at 18:04, Ben Hutcheson <ben.hutche@gmail.com>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> We do have some error codes, but it would just need
to be 
>>>>>>>>>> extended a bit. I don't think we have one for BAD
(we still 
>>>>>>>>>> receive data but the source has marked it bad).
>>>>>>>>>>
>>>>>>>>>> Timestamps I think are definitely a good idea, especially
if 
>>>>>>>>>> we eventually DNP3.
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 5, 2021 at 5:39 PM Łukasz Dywicki 
>>>>>>>>>> <luke@code-house.org>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> You have response code for each individual field
which allows 
>>>>>>>>>>> you to determine state (OK, REMOTE_ERROR etc.).
Same 
>>>>>>>>>>> information should be available also for subscriptions.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Łukasz
>>>>>>>>>>>
>>>>>>>>>>> On 05.03.2021 21:04, Andreas Vogler wrote:
>>>>>>>>>>>> Ah, forgot: and is there a status available?
>>>>>>>>>>>> Valid/invalid/good/bad
>>>>>>>>>>> indicator of a plc value?
>>>>>>>>>>>>
>>>>>>>>>>>>> On 05.03.2021, at 21:02, Andreas Vogler

>>>>>>>>>>>>> <andreas.vogler@me.com.INVALID>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I get the source time of a PlcValue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> a) from a subscription PlcSubscriptionEvent
>>>>>>>>>>>>> b) from a read request  PlcReadResponse
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Andreas
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>

Mime
View raw message