spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: EDI (Electronic Data Interchange) parser on Spark
Date Tue, 13 Mar 2018 14:16:24 GMT
Ah sorry I thought you use EDI xml.

Then you would need to build your own Spark datasource. Depending on the number of different
type of messages this will be much more or less effort.

I am not aware of any commercial or open source solution for it.

> On 13. Mar 2018, at 13:52, Aakash Basu <aakash.spark.raj@gmail.com> wrote:
> 
> Hi Jörn,
> 
> Thanks for a quick revert. I already built a EDI to JSON parser from scratch using the
811 and 820 standard mapping document. It can run on any standard and for any type of EDI.
But my built is in native python and doesn't leverage Spark's parallel processing, which I
want to do for large and huge amount of EDI data.
> 
> Any pointers on that?
> 
> Thanks,
> Aakash.
> 
>> On Tue, Mar 13, 2018 at 3:44 PM, Jörn Franke <jornfranke@gmail.com> wrote:
>> Maybe there are commercial ones. You could also some of the open source parser for
xml.
>> 
>> However xml is very inefficient and you need to du a lot of tricks to make it run
in parallel. This also depends on type of edit message etc. sophisticated unit testing and
performance testing is key.
>> 
>> Nevertheless it is also not as difficult as I made it sound now.
>> 
>> > On 13. Mar 2018, at 10:36, Aakash Basu <aakash.spark.raj@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > Did anyone built parallel and large scale X12 EDI parser to XML or JSON using
Spark?
>> >
>> > Thanks,
>> > Aakash.
> 

Mime
View raw message