samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Mangi <r...@chartbeat.com>
Subject Re: Problems upgrading Job
Date Fri, 13 Nov 2015 17:07:01 GMT
Thanks YI,

I opened SAMZA-815 <https://issues.apache.org/jira/browse/SAMZA-815>


Rick


> On Nov 13, 2015, at 2:08 AM, Yi Pan <nickpan47@gmail.com> wrote:
> 
> Hi, Rick,
> 
> Yes, please open a JIRA w/ your configuration, deployment set up and
> sequence, and logs from JobRunner.
> 
> Thanks a lot!
> 
> -Yi
> 
> On Thu, Nov 12, 2015 at 10:10 AM, Rick Mangi <rick@chartbeat.com> wrote:
> 
>> Hi Yi,
>> 
>> I pulled from master and built this morning.
>> 
>> Yes, that’s the output from JobRunner. I also tried setting a job.id to
>> see if this was an issue migrating from an old task checkpoint topic but I
>> got the same result.
>> 
>> Would you like me to open a jira ticket?
>> 
>> Thanks,
>> 
>> Rick
>> 
>> 
>> 
>>> On Nov 12, 2015, at 12:59 PM, Yi Pan <nickpan47@gmail.com> wrote:
>>> 
>>> Hi, Rick,
>>> 
>>> Did you get the fix in SAMZA-723 in your test? And could you confirm that
>>> the errors are from JobRunner log?
>>> 
>>> -Yi
>>> 
>>> On Thu, Nov 12, 2015 at 8:48 AM, Rick Mangi <rick@chartbeat.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I’m trying to migrate our samza jobs to 0.10.0 snapshot (built against
>> the
>>>> latest). Everything works fine running locally (although I had to make
>> some
>>>> changes to the local grid’s kafka since the checkpointing seems to
>> require
>>>> replication_factor > 1) but when I deploy it against my production yarn
>>>> cluster I get these errors.
>>>> 
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 ZkClient [INFO] zookeeper state
>>>> changed (SyncConnected)
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 ZkEventThread [INFO] Terminate
>>>> ZkClient event thread.
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 ZooKeeper [INFO] Session:
>>>> 0x250233cdf57f2fa closed
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 ClientCnxn [INFO] EventThread
>> shut
>>>> down
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 KafkaSystemAdmin [INFO]
>>>> Coordinator stream __samza_coordinator_metrics-reporter_1 already
>> exists.
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 JobRunner [INFO] Storing config
>> in
>>>> coordinator stream.
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 CoordinatorStreamSystemProducer
>>>> [INFO] Starting coordinator stream producer.
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 KafkaSystemProducer [INFO]
>>>> Creating a new producer for system mykafka.
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 ProducerConfig [INFO]
>>>> ProducerConfig values:
>>>> [yarnmaster01] out:     value.serializer = class
>>>> org.apache.kafka.common.serialization.ByteArraySerializer
>>>> [yarnmaster01] out:     key.serializer = class
>>>> org.apache.kafka.common.serialization.ByteArraySerializer
>>>> [yarnmaster01] out:     block.on.buffer.full = true
>>>> [yarnmaster01] out:     retry.backoff.ms = 100
>>>> [yarnmaster01] out:     buffer.memory = 33554432
>>>> [yarnmaster01] out:     batch.size = 16384
>>>> [yarnmaster01] out:     metrics.sample.window.ms = 30000
>>>> [yarnmaster01] out:     metadata.max.age.ms = 300000
>>>> [yarnmaster01] out:     receive.buffer.bytes = 32768
>>>> [yarnmaster01] out:     timeout.ms = 30000
>>>> [yarnmaster01] out:     max.in.flight.requests.per.connection = 1
>>>> [yarnmaster01] out:     bootstrap.servers = [
>>>> devstream01.chartbeat.net:9092]
>>>> [yarnmaster01] out:     metric.reporters = []
>>>> [yarnmaster01] out:     client.id =
>>>> samza_producer-metrics_reporter-1-1447342853273-4
>>>> [yarnmaster01] out:     compression.type = none
>>>> [yarnmaster01] out:     retries = 2147483647
>>>> [yarnmaster01] out:     max.request.size = 1048576
>>>> [yarnmaster01] out:     send.buffer.bytes = 131072
>>>> [yarnmaster01] out:     acks = 1
>>>> [yarnmaster01] out:     reconnect.backoff.ms = 10
>>>> [yarnmaster01] out:     linger.ms = 0
>>>> [yarnmaster01] out:     metrics.num.samples = 2
>>>> [yarnmaster01] out:     metadata.fetch.timeout.ms = 60000
>>>> [yarnmaster01] out:
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 ProducerConfig [WARN] The
>>>> configuration batch.num.messages = null was supplied but isn't a known
>>>> config.
>>>> [yarnmaster01] out: 2015-11-12 10:40:53 ProducerConfig [WARN] The
>>>> configuration producer.type = null was supplied but isn't a known
>> config.
>>>> [yarnmaster01] out: Exception in thread "main"
>>>> org.apache.samza.SamzaException:
>>>> org.apache.kafka.common.errors.TimeoutException: Failed to update
>> metadata
>>>> after 60000 ms.
>>>> [yarnmaster01] out:     at
>>>> 
>> org.apache.samza.coordinator.stream.CoordinatorStreamSystemProducer.send(CoordinatorStreamSystemProducer.java:115)
>>>> [yarnmaster01] out:     at
>>>> 
>> org.apache.samza.coordinator.stream.CoordinatorStreamSystemProducer.writeConfig(CoordinatorStreamSystemProducer.java:132)
>>>> [yarnmaster01] out:     at
>>>> org.apache.samza.job.JobRunner.run(JobRunner.scala:85)
>>>> [yarnmaster01] out:     at
>>>> org.apache.samza.job.JobRunner$.main(JobRunner.scala:43)
>>>> [yarnmaster01] out:     at
>>>> org.apache.samza.job.JobRunner.main(JobRunner.scala)
>>>> [yarnmaster01] out: Caused by:
>>>> org.apache.kafka.common.errors.TimeoutException: Failed to update
>> metadata
>>>> after 60000 ms.
>>>> [yarnmaster01] out:
>>>> 
>>>> 
>>>> Warning: run() received nonzero return code 1 while executing
>>>> './bin/run-job.sh
>>>> 
>> -config-factory=org.apache.samza.config.factories.PropertiesConfigFactory
>>>> --config-path=file://$PWD/conf/metrics_reporter.properties'!
>>>> 
>>>> 
>>>> This looks similar to https://issues.apache.org/jira/browse/SAMZA-560
>> but
>>>> I’m not using a StreamAppender in log4j.
>>>> 
>>>> Any ideas? My first thought is that I might have to delete the existing
>>>> checkpoint topics but that would mean we can’t migrate completely until
>> the
>>>> 10.0 release unless we want to run snapshot code in production.
>>>> 
>>>> Thanks!
>>>> 
>>>> Rick
>>>> 
>>>> 
>>>> 
>> 
>> 


Mime
View raw message