samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 舒琦 <sh...@eefung.com>
Subject Question about Samza Metrcis
Date Wed, 07 Sep 2016 09:09:41 GMT
Hi,

My samza job has following metrics configuration:

serializers.registry.metrics.class=org.apache.samza.serializers.MetricsSnapshotSerdeFactory

systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
systems.kafka.consumer.zookeeper.connect=zk11:3181,zk12:3181,zk13:3181
systems.kafka.producer.bootstrap.servers=buka1:9096,buka2:9096,buka3:9096

systems.kafka.streams.samza-metrics.samza.msg.serde=metrics

metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory
metrics.reporter.snapshot.stream=kafka.samza-metrics
metrics.reporters=snapshot

And the job is deployed on yarn, after job started, everything is fine, I also can find coordinator
stream and checkpoint stream in the same kafka cluster, but there is no samza-metrics stream.

One of the container log :

2016-09-07 16:32:31.947 [main] MetricsSnapshotReporterFactory [WARN] Unable to find implementation
version in jar's meta info. Defaulting to 0.0.1.
2016-09-07 16:32:31.948 [main] MetricsSnapshotReporterFactory [INFO] Got system stream SystemStream
[system=kafka, stream=samza-metrics].
2016-09-07 16:32:31.949 [main] MetricsSnapshotReporterFactory [INFO] Got system factory org.apache.samza.system.kafka.KafkaSystemFactory@1eed1f10.
2016-09-07 16:32:31.950 [main] MetricsSnapshotReporterFactory [INFO] Got producer org.apache.samza.system.kafka.KafkaSystemProducer@16d96b45.
2016-09-07 16:32:31.951 [main] MetricsSnapshotReporterFactory [INFO] Got serde org.apache.samza.serializers.MetricsSnapshotSerde@569f129d.
2016-09-07 16:32:31.952 [main] MetricsSnapshotReporterFactory [INFO] Setting polling interval
to 60
2016-09-07 16:32:31.954 [main] MetricsSnapshotReporter [INFO] got metrics snapshot reporter
properties [job name: data-status-persistent-hstore, job id: 1, containerName: samza-container-1,
version: 0.0.1, samzaVersion: 0.10.1, host: store116, pollingInterval 60]
2016-09-07 16:32:31.955 [main] MetricsSnapshotReporter [INFO] Registering MetricsSnapshotReporterFactory
with producer.
2016-09-07 16:32:31.955 [main] SamzaContainer$ [INFO] Got metrics reporters: Set(snapshot)

2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO] Registering TaskName-Partition
7 with producer.
2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO] Registering TaskName-Partition
1 with producer.
2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO] Registering TaskName-Partition
5 with producer.
2016-09-07 16:32:32.016 [main] MetricsSnapshotReporter [INFO] Registering TaskName-Partition
3 with producer.
2016-09-07 16:32:32.017 [main] SamzaContainer [INFO] Starting JVM metrics.
2016-09-07 16:32:32.017 [main] SamzaContainer [INFO] Starting metrics reporters.
2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO] Registering samza-container-1
with producer.
2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO] Starting producer.
2016-09-07 16:32:32.018 [main] MetricsSnapshotReporter [INFO] Starting reporter timer.
2016-09-07 16:32:32.019 [main] SamzaContainer [INFO] Registering task instances with offsets.
2016-09-07 16:32:32.022 [main] SamzaContainer [INFO] Starting offset manager.

2016-09-07 16:32:32.212 [SAMZA-METRIC-SNAPSHOT-REPORTER] KafkaSystemProducer [INFO] Creating
a new producer for system kafka.
2016-09-07 16:32:32.221 [SAMZA-METRIC-SNAPSHOT-REPORTER] ProducerConfig [INFO] ProducerConfig
values:
       	value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
       	key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
       	block.on.buffer.full = true
       	retry.backoff.ms = 100
       	buffer.memory = 33554432
       	batch.size = 16384
       	metrics.sample.window.ms = 30000
       	metadata.max.age.ms = 300000
       	receive.buffer.bytes = 32768
       	timeout.ms = 30000
       	max.in.flight.requests.per.connection = 1
       	bootstrap.servers = [buka1:9096, buka2:9096, buka3:9096]
       	metric.reporters = []
       	client.id = samza_producer-data_status_persistent_hstore-1-1473237151949-4
       	compression.type = none
       	retries = 2147483647
       	max.request.size = 1048576
       	send.buffer.bytes = 131072
       	acks = 1
       	reconnect.backoff.ms = 10
       	linger.ms = 0
       	metrics.num.samples = 2
       	metadata.fetch.timeout.ms = 60000

Thanks.

————————
舒琦
地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
网址:http://www.eefung.com
微博:http://weibo.com/eefung
邮编:410013
电话:400-677-0986
传真:0731-88519609


Mime
View raw message