storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivpriya Tamboskar <Shivpriya_Tambos...@symantec.com>
Subject Re: Storm-HDFS bolt file paths according to input Tuples
Date Mon, 04 Jan 2016 22:04:21 GMT
Thank you Aaron,
 I too did try to check the code to see if there is any way to modify the code and get this
feature.
But it seemed too intricate. I was trying to get an idea if there are any possible solutions
 the other users came up with in this situation.

-Shivpriya

From: "Aaron.Dossett" <Aaron.Dossett@target.com<mailto:Aaron.Dossett@target.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Date: Monday, January 4, 2016 at 1:49 PM
To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Subject: Re: Storm-HDFS bolt file paths according to input Tuples

Shivpriya,

I have thought about solving that same problem, and there is no easy answer in the current
code that I could see.  You would have to modify hdfs-bolt to track a group of file handles
into HDFS and define a mechanism to map a tuple to the corresponding file handle, create it
if it doesn’t already exist, etc.  And you have to do it in a way that works for all of
the existing flavors of hdfs-bolt (text files, sequence files, avro files).  Now hive-bolt
does something conceptually similar in that in manages a connection pool, limits maximum connections,
closes old connections, etc. so the pattern does exist but implementing it would be tantamount
to a rewrite of hdfs-bolt.

For the problem I was trying to solve, this approach would have provided only a very small
benefit, so I didn’t pursue it beyond some high level design work.  You could open a JIRA
for this feature request.

Thanks, Aaron

From: Shivpriya Tamboskar <Shivpriya_Tamboskar@symantec.com<mailto:Shivpriya_Tamboskar@symantec.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Date: Monday, January 4, 2016 at 3:40 PM
To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Subject: Storm-HDFS bolt file paths according to input Tuples

Hello,
  I am trying to write data into HDFS using the storm-hdfs bolt http://storm.apache.org/documentation/storm-hdfs.html

The problem I am trying to solve is that I want the file paths to be set according to the
input tuples.


FileNameFormat fileNameFormat = new DefaultFileNameFormat()

        .withPath("/foo/“);


I am getting inputs from various users and what want to do is set the file paths according
to the username e.g.  "/user1/“ , “/user2/“


As far as I know once the path is set for the bolt it can’t be changed unless you have file
rotation policy set which is not what I want to do.


Has anyone else tried to solve this before ? Any suggestions would be appreciated.


Thanks

Shivpriya



Mime
View raw message