drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aman Sinha <asi...@maprtech.com>
Subject Re: Drill dir0 issue
Date Sun, 23 Aug 2015 19:45:00 GMT
Yes, I just realized that and was about to respond to my prior message.
I just tested with a directory structure similar to Sungwook's  (where
directories are named with 'year=2012' format) and it works for me.
But I am on the current master branch.
In the original message 'Sometimes it picks up e.g., year=2010, but not
year=2012..'   that clearly sounds like wrong result...
definitely file a JIRA with a repro.

Aman

On Sun, Aug 23, 2015 at 12:23 PM, Jacques Nadeau <jacques@dremio.com> wrote:

> The way that Sungwook is describing the issue, it has nothing to do with
> Hive.  The files were generated via Hive but he is querying directly
> through the DFS schema.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Sun, Aug 23, 2015 at 12:20 PM, Aman Sinha <asinha@maprtech.com> wrote:
>
> > Sungwook, do you have the latest master build which has the fix for Hive
> > partition pruning (DRILL-3121) ?
> >
> > On Sun, Aug 23, 2015 at 12:15 PM, Sungwook Yoon <syoon@maprtech.com>
> > wrote:
> >
> > > Will do,
> > >
> > > Thanks,
> > >
> > > Sungwook
> > >
> > >
> > > On Sun, Aug 23, 2015 at 2:14 PM, Jacques Nadeau <jacques@dremio.com>
> > > wrote:
> > >
> > > > It sounds like a bug. Can you file a jira?
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Sun, Aug 23, 2015 at 12:13 PM, Sungwook Yoon <syoon@maprtech.com>
> > > > wrote:
> > > >
> > > > > Hi Jacques,
> > > > >
> > > > > This works well, no problem of accessing the partitioned dirs.
> > > > > (and actually pretty faster than accessing from one level above)
> > > > >
> > > > > Just the issues I asked about, when I access from the
> > > > > /user/hive/warehouse/table, it somehow does not recover every dir0.
> > > > >
> > > > > Sungwook
> > > > >
> > > > >
> > > > > On Sun, Aug 23, 2015 at 2:02 PM, Jacques Nadeau <
> jacques@dremio.com>
> > > > > wrote:
> > > > >
> > > > > > I think Hsuan misunderstood your question.
> > > > > >
> > > > > > Can you let us know what you get if you query:
> > > > > >
> > > > > > select * from dfs.root.`/user/hive/warehouse/table/year=2012`
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Jacques Nadeau
> > > > > > CTO and Co-Founder, Dremio
> > > > > >
> > > > > > On Sun, Aug 23, 2015 at 7:07 AM, Sungwook Yoon <
> syoon@maprtech.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I am trying to use Hive parquet stored files partitioned
by
> some
> > > > > column.
> > > > > > > So, the directory structure is partitioned with the column.
> > > > > > >
> > > > > > > The column is actually year.
> > > > > > > Let's say there are 5 years, so dir0 are like year=2010,
> > > > > > > year=2011,year=2012,year=2013,year=2014
> > > > > > >
> > > > > > > We did like following
> > > > > > > select * from dfs.root.`/user/hive/warehouse/table` d where
> > d.dir0
> > > =
> > > > > > > 'year=2012';
> > > > > > >
> > > > > > > I get nothing.
> > > > > > > Apparently, there are parquet files in the directory though.
> > > > > > >
> > > > > > > Sometimes it picks up e.g., year=2010, but not year=2012..
> > > > > > >
> > > > > > > Where am I going wrong with this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Sungwook
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message