pig-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Atul Raut <atul.r...@indicussoftware.com>
Subject Re: How to execute pig script from java
Date Mon, 13 Aug 2018 03:52:04 GMT
Hello Rohini,

Thank you for your reply,

Please find following log,

2018-08-13 09:19:18,644 -- DEBUG -- address: Acamar/192.168.102.189 
isLoopbackAddress: false, with host 192.168.102.189 Acamar
2018-08-13 09:19:19,509 -- WARN -- Your hostname, Acamar resolves to a 
loopback/non-reachable address: fe80:0:0:0:d53f:fff5:c5cd:d591%eth10, 
but we couldn't find any external IP address!
2018-08-13 09:19:19,633 -- DEBUG -- Using SLF4J as the default logging 
framework
2018-08-13 09:19:20,137 -- DEBUG -- Create a new graph.
2018-08-13 09:19:20,436 -- DEBUG -- Original macro AST:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))))

2018-08-13 09:19:20,436 -- DEBUG -- macro AST after import:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))))

2018-08-13 09:19:20,436 -- DEBUG -- Resulting macro AST:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))))

2018-08-13 09:19:21,379 -- DEBUG -- Could not find schema file for 
/apps/hive/warehouse/cnx.db/employee/atul.csv
2018-08-13 09:19:21,384 -- DEBUG -- Original macro AST:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))) 
(STATEMENT (STORE A '/user/Admin/pig_Output1/' (FUNC PigStorage))))

2018-08-13 09:19:21,385 -- DEBUG -- macro AST after import:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))) 
(STATEMENT (STORE A '/user/Admin/pig_Output1/' (FUNC PigStorage))))

2018-08-13 09:19:21,385 -- DEBUG -- Resulting macro AST:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))) 
(STATEMENT (STORE A '/user/Admin/pig_Output1/' (FUNC PigStorage))))

2018-08-13 09:19:21,400 -- DEBUG -- Could not find schema file for 
/apps/hive/warehouse/cnx.db/employee/atul.csv
2018-08-13 09:19:21,622 -- DEBUG -- Original macro AST:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))) 
(STATEMENT (STORE A '/user/Admin/pig_Output1/' (FUNC PigStorage))))

2018-08-13 09:19:21,622 -- DEBUG -- macro AST after import:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))) 
(STATEMENT (STORE A '/user/Admin/pig_Output1/' (FUNC PigStorage))))

2018-08-13 09:19:21,622 -- DEBUG -- Resulting macro AST:
(QUERY (STATEMENT A (LOAD 
'/apps/hive/warehouse/cnx.db/employee/atul.csv' (FUNC PigStorage))) 
(STATEMENT (STORE A '/user/Admin/pig_Output1/' (FUNC PigStorage))))

2018-08-13 09:19:21,637 -- DEBUG -- Could not find schema file for 
/apps/hive/warehouse/cnx.db/employee/atul.csv
2018-08-13 09:19:21,642 -- DEBUG -- Pig Internal storage in use
2018-08-13 09:19:21,731 -- INFO -- Pig features used in the script: UNKNOWN
2018-08-13 09:19:21,769 -- INFO -- Key [pig.schematuple] was not set... 
will not generate code.
2018-08-13 09:19:21,807 -- INFO -- {RULES_ENABLED=[AddForEach, 
ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, 
LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, 
NestedLimitOptimizer, PartitionFilterOptimizer, 
PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, 
SplitFilter, StreamTypeCastInserter]}
2018-08-13 09:19:21,960 -- DEBUG -- Found heap (Code Cache) of type 
Non-heap memory
2018-08-13 09:19:21,960 -- DEBUG -- Found heap (Metaspace) of type 
Non-heap memory
2018-08-13 09:19:21,960 -- DEBUG -- Found heap (Compressed Class Space) 
of type Non-heap memory
2018-08-13 09:19:21,960 -- DEBUG -- Found heap (PS Eden Space) of type 
Heap memory
2018-08-13 09:19:21,961 -- DEBUG -- Found heap (PS Survivor Space) of 
type Heap memory
2018-08-13 09:19:21,961 -- DEBUG -- Found heap (PS Old Gen) of type Heap 
memory
2018-08-13 09:19:21,961 -- INFO -- Selected heap (PS Old Gen) of size 
1404043264 to monitor. collectionUsageThreshold = 1037041664, 
usageThreshold = 1037041664
2018-08-13 09:19:22,046 -- INFO -- File concatenation threshold: 100 
optimistic? false
2018-08-13 09:19:22,070 -- DEBUG -- Not a sampling job.
2018-08-13 09:19:22,086 -- DEBUG -- Cannot find POLocalRearrange or 
POUnion in map leaf, skip secondary key optimizing
2018-08-13 09:19:22,093 -- INFO -- MR plan size before optimization: 1
2018-08-13 09:19:22,093 -- INFO -- MR plan size after optimization: 1
2018-08-13 09:19:22,189 -- INFO -- Pig script settings are added to the job
2018-08-13 09:19:22,197 -- INFO -- 
mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2018-08-13 09:19:22,260 -- INFO -- This job cannot be converted run 
in-process
Exception in thread "main" java.lang.UnsatisfiedLinkError: 
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
     at 
org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native 
Method)
     at 
org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86)
     at 
org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:430)
     at 
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:202)
     at 
org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:163)
     at 
org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:144)
     at 
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2250)
     at 
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2232)
     at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
     at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
     at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
     at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:261)
     at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
     at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.shipToHDFS(JobControlCompiler.java:1809)
     at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.putJarOnClassPathThroughDistributedCache(JobControlCompiler.java:1689)
     at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:659)
     at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:331)
     at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:221)
     at 
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
     at org.apache.pig.PigServer.launchPlan(PigServer.java:1475)
     at 
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1460)
     at org.apache.pig.PigServer.storeEx(PigServer.java:1119)
     at org.apache.pig.PigServer.store(PigServer.java:1082)
     at org.apache.pig.PigServer.openIterator(PigServer.java:995)
     at com.contineonx.ext.pig.Test_Atul.main(Test_Atul.java:29)


Regards,
Atul Raut

On 8/11/2018 2:11, Rohini Palaniswamy wrote:
> What you are missing in your source code is call to openIterator() or 
> store() to retrieve the results.
>
> http://pig.apache.org/docs/r0.17.0/api/org/apache/pig/PigServer.html 
> <http://pig.apache.org/docs/r0.17.0/api/org/apache/pig/PigServer.html>
> The programmer then registers queries using registerQuery() and 
> retrieves results using openIterator() or store()
>
>
>
> On Fri, Aug 10, 2018 at 1:36 PM, Rohini Palaniswamy <rohini@apache.org 
> <mailto:rohini@apache.org>> wrote:
>
>     Pig does not have any server. The client directly launches jobs on
>     the YARN cluster. You can just use the APIs in
>     http://pig.apache.org/docs/r0.17.0/api/org/apache/pig/PigServer.html
>     <http://pig.apache.org/docs/r0.17.0/api/org/apache/pig/PigServer.html>
>     to execute scripts from your java program.
>
>     On Sun, Jul 29, 2018 at 8:24 PM, Atul Raut <rautatul44@gmail.com
>     <mailto:rautatul44@gmail.com>> wrote:
>
>         How to execute pig script from java class.
>
>         I need to submit Pig script from my java application to the
>         Pig server
>         (This may be on any remote location) and that Pig server will
>         execute that
>         script and return the result to my java application.
>
>
>
>         Following is my source,
>
>         public static void main(String[] args) throws Exception
>         { System.setProperty("hadoop.home.dir",
>         "/Pig/hadoop-common-2.2.0-bin-master/"); Properties props = new
>         Properties(); props.setProperty("fs.default.name
>         <http://fs.default.name>", "hdfs://
>         192.168.102.179:8020 <http://192.168.102.179:8020>");
>         props.setProperty("mapred.job.tracker", "
>         192.168.102.179:8021 <http://192.168.102.179:8021>");
>         props.setProperty("pig.use.overriden.hadoop.configs", "true");
>         PigServer
>         pig = new PigServer(ExecType.MAPREDUCE, props); pig.debugOn();
>         pig.registerScript("A = LOAD '/apps/employee/sample.txt' USING
>         PigStorage();"); }
>
>
>
>         Regards,
>         Atul Raut
>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message