beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Al Yaros (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure
Date Wed, 06 Sep 2017 13:20:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155311#comment-16155311
] 

Al Yaros commented on BEAM-2837:
--------------------------------

Hey,
I didn't saw anything else in the error.
I also think that the reason is dependencies issue but I have tried all kind of combination
of versions (Include other and Exclude others) but still it refused to work.
You can look on the GitHub code (Link added) that this is very simple pipeline that even if
the pom only includes:

{code:java}
 <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
            <version>2.1.0</version>
</dependency>
{code}

It doesn't work.

I verified that the Project\Instance\DB\Table\Columns all exists in Spanner.
I'm running it after I ran "gcloud auth application-default login" from the Cloud SDK Console
so It's not a manner of Authentication. (It starts the pipeline in DataFlow)
I have tried to Exclude Spanner from the "beam-runners-google-cloud-dataflow-java" and include
the official spanner Java driver (in different versions) and still it didn't worked.
Anyway, I'll be happy if you can share an example (Simple Code, Pom Etc..) or Versions I should
use to be able to write to Spanner from GCP DataFlow.
Thanks


> Writing To Spanner From Google Cloud DataFlow - Failure
> -------------------------------------------------------
>
>                 Key: BEAM-2837
>                 URL: https://issues.apache.org/jira/browse/BEAM-2837
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow
>    Affects Versions: 2.1.0
>         Environment: Google Cloud DataFlow
>            Reporter: Al Yaros
>            Assignee: Mairbek Khadikov
>
> Simple Java Program That reads from Pub\Sub and Writes to Spanner Fails with cryptic
error message.
> Simple Program to Demonstrate the Error:
> [https://github.com/alyaros/ExamplePubSubToSpannerViaDataFlow]
> {code:java}
> *Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError:
Could not initialize class com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
>         org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
>         org.apache.beam.sdk.io.
> gcp.spanner.SpannerWriteGroupFn$DoFnInvoker.invokeSetup(Unknown Source)
>         com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:66)
>         com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:48)
>         com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:104)
>         com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:66)
>         com.google.cloud.dataflow.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:360)
>         com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:271)
>         com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:253)
>         com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
>         com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
>         com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
>         com.google.cloud.dataflow.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:142)
>         com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:925)
>         com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
>         com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         java.lang.Thread.run(Thread.java:745)*
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message