spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com>
Subject Re: OOM for HiveFromSpark example
Date Wed, 25 Mar 2015 17:54:28 GMT
I am facing same issue, posted a new thread. Please respond.

On Wed, Jan 14, 2015 at 4:38 AM, Zhan Zhang <zzhang@hortonworks.com> wrote:

> Hi Folks,
>
> I am trying to run hive context in yarn-cluster mode, but met some error.
> Does anybody know what cause the issue.
>
> I use following cmd to build the distribution:
>
>  ./make-distribution.sh -Phive -Phive-thriftserver  -Pyarn  -Phadoop-2.4
>
> 15/01/13 17:59:42 INFO cluster.YarnClusterScheduler:
> YarnClusterScheduler.postStartHook done
> 15/01/13 17:59:42 INFO storage.BlockManagerMasterActor: Registering block
> manager cn122-10.l42scl.hortonworks.com:56157 with 1589.8 MB RAM,
> BlockManagerId(2, cn122-10.l42scl.hortonworks.com, 56157)
> 15/01/13 17:59:43 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF
> NOT EXISTS src (key INT, value STRING)
> 15/01/13 17:59:43 INFO parse.ParseDriver: Parse Completed
> 15/01/13 17:59:44 INFO metastore.HiveMetaStore: 0: Opening raw store with
> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> 15/01/13 17:59:44 INFO metastore.ObjectStore: ObjectStore, initialize
> called
> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property
> datanucleus.cache.level2 unknown - will be ignored
> 15/01/13 17:59:44 INFO DataNucleus.Persistence: Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not
> present in CLASSPATH (or one of dependencies)
> 15/01/13 17:59:44 WARN DataNucleus.Connection: BoneCP specified but not
> present in CLASSPATH (or one of dependencies)
> 15/01/13 17:59:52 INFO metastore.ObjectStore: Setting MetaStore object pin
> classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 15/01/13 17:59:52 INFO metastore.MetaStoreDirectSql: MySQL check failed,
> assuming we are not on mysql: Lexical error at line 1, column 5.
> Encountered: "@" (64), after : "".
> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/01/13 17:59:53 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/01/13 17:59:59 INFO DataNucleus.Datastore: The class
> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
> "embedded-only" so does not have its own datastore table.
> 15/01/13 18:00:00 INFO metastore.ObjectStore: Initialized ObjectStore
> 15/01/13 18:00:00 WARN metastore.ObjectStore: Version information not
> found in metastore. hive.metastore.schema.verification is not enabled so
> recording the schema version 0.13.1aa
> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added admin role in
> metastore
> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: Added public role in
> metastore
> 15/01/13 18:00:01 INFO metastore.HiveMetaStore: No user is added in admin
> role, since config is empty
> 15/01/13 18:00:01 INFO session.SessionState: No Tez session required at
> this point. hive.execution.engine=mr.
> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=Driver.run
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=TimeToSubmit
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:02 INFO ql.Driver: Concurrency mode is disabled, not
> creating a lock manager
> 15/01/13 18:00:02 INFO log.PerfLogger: <PERFLOG method=compile
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=parse
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO parse.ParseDriver: Parsing command: CREATE TABLE IF
> NOT EXISTS src (key INT, value STRING)
> 15/01/13 18:00:03 INFO parse.ParseDriver: Parse Completed
> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=parse
> start=1421190003030 end=1421190003031 duration=1
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=semanticAnalyze
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
> 15/01/13 18:00:03 INFO parse.SemanticAnalyzer: Creating table src
> position=27
> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_table : db=default
> tbl=src
> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang
> ip=unknown-ip-addr      cmd=get_table : db=default tbl=src
> 15/01/13 18:00:03 INFO metastore.HiveMetaStore: 0: get_database: default
> 15/01/13 18:00:03 INFO HiveMetaStore.audit: ugi=zzhang
> ip=unknown-ip-addr      cmd=get_database: default
> 15/01/13 18:00:03 INFO ql.Driver: Semantic Analysis Completed
> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=semanticAnalyze
> start=1421190003031 end=1421190003406 duration=375
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO ql.Driver: Returning Hive schema:
> Schema(fieldSchemas:null, properties:null)
> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=compile
> start=1421190002998 end=1421190003416 duration=418
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=Driver.execute
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO ql.Driver: Starting command: CREATE TABLE IF NOT
> EXISTS src (key INT, value STRING)
> 15/01/13 18:00:03 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit
> start=1421190002995 end=1421190003421 duration=426
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=runTasks
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO log.PerfLogger: <PERFLOG method=task.DDL.Stage-0
> from=org.apache.hadoop.hive.ql.Driver>
> 15/01/13 18:00:03 INFO exec.DDLTask: Default to LazySimpleSerDe for table
> src
> 15/01/13 18:00:05 INFO log.PerfLogger: </PERFLOG method=Driver.execute
> start=1421190003416 end=1421190005498 duration=2082
> from=org.apache.hadoop.hive.ql.Driver>
> Exception in thread "Driver"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Driver"
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
Deepak

Mime
View raw message