samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Zuljevic <r.zulje...@levi9.com>
Subject Running Samza programmatically
Date Wed, 09 Mar 2016 11:47:05 GMT
Hi all,

We would like to write and run integration tests for our own Samza Tasks/Jobs, but without
using external tools like Zoopkio, but rather as a part of the build process. What we would
like to do is start Yarn/Zookeeper/Kafka locally (or in a docker container) and then submit
jobs programmatically (ie bypassing run-job.sh script) in order to have a bit more control
over status and life of jobs. This is how we are trying to start jobs currently:

CommandLine cl = new CommandLine();
OptionSet options = cl.parser().parse("--config-factory=org.apache.samza.config.factories.PropertiesConfigFactory",
                                                                "--config-path=" + App.class.getResource("/config/test.properties").getPath());
Config config = new MapConfig(configMap);
JobRunner runner = new JobRunner(cl.loadConfig(options));

The job is submitted successfully, but it immediately fails and no logs are produced. The
config we are using is the standard hello-samza one, with the caveat that yarn.package.path
property is set to the absolute path where the packaged *.tar.gz file resides.

I know that run-class.sh adds a lot of dependencies to the class-path and sets up some variables,
should we try to mimic this behavior as well, or is there a more straightforward approach?

Thank you for your response!

Met vriendelijke groet / Kind regards,
Robert Žuljević
Software Developer
[Title: Levi9 IT Services]
________________________________
Address: Trifkovicev trg 6, 21000 Novi Sad, Serbia
Tel.: +31 20 6701 947 | +381 21 2155 500
Mobile: +381 64 428 28 46
Skype: robert.zuljevic
Internet: www.levi9.com<http://www.levi9.com/>

Chamber of commerce Levi9 Holding: 34221951
Chamber of commerce Levi9 IT Services BV: 34224746
________________________________
This e-mail may contain confidential or privileged information. If you are not (one of) the
intended recipient(s), please notify the sender immediately by reply e-mail and delete this
message and any attachments permanently without retaining a copy. Any review, disclosure,
copying, distribution or taking any action in reliance on the contents of this e-mail by persons
or entities other than the intended recipient(s) is strictly prohibited and may be unlawful.
The services of Levi9 are exclusively subject to its general terms and conditions. These general
terms and conditions can be found on www.levi9.com<http://www.levi9.com/> and a copy
will be promptly submitted to you on your request and free of charge.


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message