samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navina Ramesh <nram...@linkedin.com.INVALID>
Subject Re: Question about Samza Metrcis
Date Thu, 08 Sep 2016 20:19:18 GMT
Hi ShuQi,

Auto-creation of streams depends on your Kafka server configuration. In
case of coordinator stream and checkpoint stream, samza explicitly creates
a stream with 1 partition before publishing to it. This doesn't apply for
metrics. So, if auto-creation is turned off in kafka server, then you have
to manually create it, as you did.

Glad you figured it out.

Cheers!
Navina

On Wed, Sep 7, 2016 at 6:37 PM, 舒琦 <shuqi@eefung.com> wrote:

> Hi,
>
> Thanks for your help.
>
> I think checkpoint stream and coordinate stream will be auto-create per
> job if using kaka, but not metrics.
>
> After I manually created metrics stream in Kafka, the metrics is written
> into the stream.
>
>
> ShuQi
>
> > 在 2016年9月7日,23:15,Jagadish Venkatraman <jagadish1989@gmail.com>
写道:
> >
> > Can you run your program in DEBUG log-level? Does sending the metric to
> the
> > producer fail? Is the metric reporter thread showing an exception? (check
> > the stderr file too)
> >
> > Producing to a kafka topic should usually auto-create it.
> >
> > On Wed, Sep 7, 2016 at 2:09 AM, 舒琦 <shuqi@eefung.com> wrote:
> >
> >> Hi,
> >>
> >> My samza job has following metrics configuration:
> >>
> >> serializers.registry.metrics.class=org.apache.samza.serializers.
> >> MetricsSnapshotSerdeFactory
> >>
> >> systems.kafka.samza.factory=org.apache.samza.system.kafka.
> >> KafkaSystemFactory
> >> systems.kafka.consumer.zookeeper.connect=zk11:3181,zk12:3181,zk13:3181
> >> systems.kafka.producer.bootstrap.servers=buka1:9096,
> buka2:9096,buka3:9096
> >>
> >> systems.kafka.streams.samza-metrics.samza.msg.serde=metrics
> >>
> >> metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.
> >> MetricsSnapshotReporterFactory
> >> metrics.reporter.snapshot.stream=kafka.samza-metrics
> >> metrics.reporters=snapshot
> >>
> >> And the job is deployed on yarn, after job started, everything is fine,
> I
> >> also can find coordinator stream and checkpoint stream in the same kafka
> >> cluster, but there is no samza-metrics stream.
> >>
> >> One of the container log :
> >>
> >> 2016-09-07 16:32:31.947 [main] MetricsSnapshotReporterFactory [WARN]
> >> Unable to find implementation version in jar's meta info. Defaulting to
> >> 0.0.1.
> >> 2016-09-07 16:32:31.948 [main] MetricsSnapshotReporterFactory [INFO] Got
> >> system stream SystemStream [system=kafka, stream=samza-metrics].
> >> 2016-09-07 16:32:31.949 [main] MetricsSnapshotReporterFactory [INFO] Got
> >> system factory org.apache.samza.system.kafka.
> KafkaSystemFactory@1eed1f10.
> >> 2016-09-07 16:32:31.950 [main] MetricsSnapshotReporterFactory [INFO] Got
> >> producer org.apache.samza.system.kafka.KafkaSystemProducer@16d96b45.
> >> 2016-09-07 16:32:31.951 [main] MetricsSnapshotReporterFactory [INFO] Got
> >> serde org.apache.samza.serializers.MetricsSnapshotSerde@569f129d.
> >> 2016-09-07 16:32:31.952 [main] MetricsSnapshotReporterFactory [INFO]
> >> Setting polling interval to 60
> >> 2016-09-07 16:32:31.954 [main] MetricsSnapshotReporter [INFO] got
> metrics
> >> snapshot reporter properties [job name: data-status-persistent-hstore,
> job
> >> id: 1, containerName: samza-container-1, version: 0.0.1, samzaVersion:
> >> 0.10.1, host: store116, pollingInterval 60]
> >> 2016-09-07 16:32:31.955 [main] MetricsSnapshotReporter [INFO]
> Registering
> >> MetricsSnapshotReporterFactory with producer.
> >> 2016-09-07 16:32:31.955 [main] SamzaContainer$ [INFO] Got metrics
> >> reporters: Set(snapshot)
> >>
> >> 2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO]
> Registering
> >> TaskName-Partition 7 with producer.
> >> 2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO]
> Registering
> >> TaskName-Partition 1 with producer.
> >> 2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO]
> Registering
> >> TaskName-Partition 5 with producer.
> >> 2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO]
> Registering
> >> TaskName-Partition 3 with producer.
> >> 2016-09-07 16:32:32.017 [main] SamzaContainer [INFO] Starting JVM
> metrics.
> >> 2016-09-07 16:32:32.017 [main] SamzaContainer [INFO] Starting metrics
> >> reporters.
> >> 2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO]
> Registering
> >> samza-container-1 with producer.
> >> 2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO] Starting
> >> producer.
> >> 2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO] Starting
> >> reporter timer.
> >> 2016-09-07 16:32:32.019 [main] SamzaContainer [INFO] Registering task
> >> instances with offsets.
> >> 2016-09-07 16:32:32.022 [main] SamzaContainer [INFO] Starting offset
> >> manager.
> >>
> >> 2016-09-07 16:32:32.212 [SAMZA-METRIC-SNAPSHOT-REPORTER]
> >> KafkaSystemProducer [INFO] Creating a new producer for system kafka.
> >> 2016-09-07 16:32:32.221 [SAMZA-METRIC-SNAPSHOT-REPORTER] ProducerConfig
> >> [INFO] ProducerConfig values:
> >>        value.serializer = class org.apache.kafka.common.serialization.
> >> ByteArraySerializer
> >>        key.serializer = class org.apache.kafka.common.serialization.
> >> ByteArraySerializer
> >>        block.on.buffer.full = true
> >>        retry.backoff.ms = 100
> >>        buffer.memory = 33554432
> >>        batch.size = 16384
> >>        metrics.sample.window.ms = 30000
> >>        metadata.max.age.ms = 300000
> >>        receive.buffer.bytes = 32768
> >>        timeout.ms = 30000
> >>        max.in.flight.requests.per.connection = 1
> >>        bootstrap.servers = [buka1:9096, buka2:9096, buka3:9096]
> >>        metric.reporters = []
> >>        client.id = samza_producer-data_status_persistent_hstore-1-
> >> 1473237151949-4
> >>        compression.type = none
> >>        retries = 2147483647
> >>        max.request.size = 1048576
> >>        send.buffer.bytes = 131072
> >>        acks = 1
> >>        reconnect.backoff.ms = 10
> >>        linger.ms = 0
> >>        metrics.num.samples = 2
> >>        metadata.fetch.timeout.ms = 60000
> >>
> >> Thanks.
> >>
> >> ————————
> >> 舒琦
> >> 地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
> >> 网址:http://www.eefung.com
> >> 微博:http://weibo.com/eefung
> >> 邮编:410013
> >> 电话:400-677-0986
> >> 传真:0731-88519609
> >>
> >>
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
>
>


-- 
Navina R.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message