spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Livni, Dana" <dana.li...@intel.com>
Subject working with MultiTableInputFormat
Date Sat, 29 Mar 2014 07:42:47 GMT
I'm trying to create an RDD from multiple scans.
I tried to set the configuration this way:

Configuration config = HBaseConfiguration.create();
config.setStrings(MultiTableInputFormat.SCANS,scanStrings);

And creating each scan string in the array scanStrings  this way:

Scan scan = new Scan();
scan.setAttribute(Scan.SCAN_ATTRIBUTES_TABLE_NAME, Bytes.toBytes(tableName));
scan.setFilter(filter);
ByteArrayOutputStream out = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(out);
scan.write(dos);
String singleScanString = Base64.encodeBytes(out.toByteArray());
*         When doing so I got an exception of  "No table was provided" . From the class TableInputFormatBase
*         Even it didn't make any seance to me cause I'm providing the input table in the
attribute SCAN_ATTRIBUTES_TABLE_NAME
*         I tried adding config.set(TableInputFormat.INPUT_TABLE, tableName); to my configuration
*         But then my spark mapper run into some kind of infinity loop.
Do I miss anything?
Can spark work with MultiTableInputFormat or only with TableInputFormat?

Thanks Dana.

---------------------------------------------------------------------
Intel Electronics Ltd.

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Mime
View raw message