drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Bates <jba...@maprtech.com>
Subject How do I make json files les painful
Date Thu, 19 Mar 2015 16:22:24 GMT
I constantly, constantly, constantly hit this.

I have json files that are just a huge collection of an array of json
objects

example
"MyArrayInTheFile":
[{"a":"1","b":"2","c":"3"},{"a":"1","b":"2","c":"3"},...]

My issue is in exploring the data, I hit this.

Query failed: Query stopped., Record was too large to copy into vector. [
39186288-2e01-408c-b886-dcee0a2c25c5 on maprdemo:31010 ]

I can explore csv, tab, maprdb, hive at fairly large data sets and limit
the response to what fits in my system limitations but not json in this
format.

The two options I have come up with to move forward are..

   1. I strip out 90% of the array values in a file and explore that to get
   to my view. then go to a larger system and see if I have enough to get the
   job done.
   2. Move to the larger system and explore there taking resources that
   don't need to be spent on a science project.

Hoping the smart people have a different option for me,

Jim

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message