crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wu lihu <>
Subject Re: How to deal with the log files end with gz compressed
Date Thu, 22 Sep 2016 13:04:11 GMT
Oh... I forgot the Crunch is only an abstract for MapReduce pipeline.
But anyone tried use it with S3 job output ?  It's strange, seems the
job froze after write the _SUCESS output to S3. The last log appeared
in my job log file is like below:

2016-09-22 10:05:37,194 INFO
(Thread-5): Job status available at:

2016-09-22 10:12:13,692 INFO
(Thread-5): close closed:false

2016-09-22 1:09 GMT+08:00 Josh Wills <>:
> I don't follow- Hadoop handles compression transparently for most of the
> commonly used input formats and compression schemes; you shouldn't have to
> do anything.
> On Wed, Sep 21, 2016 at 12:53 AM wu lihu <> wrote:
>> Hi Everyone
>>   I want to ask one question about process the logs files end with
>> compressed files ? Is there any example for that ?

View raw message