spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Praveen Devarao" <praveen...@in.ibm.com>
Subject Re: Guidelines for writing SPARK packages
Date Tue, 02 Feb 2016 09:20:40 GMT
Thanks David.

I am looking at extending the SparkSQL library with a custom 
package...hence was looking at more from details on any specific classes 
to be extended or implement (with) to achieve the redirect of calls to my 
module (when using .format).

If you have any info on these lines do share with me...else debugging 
through would be the way :-)

Thanking You

Praveen Devarao



From:   David Russell <themarchoffolly@gmail.com>
To:     Praveen Devarao/India/IBM@IBMIN
Cc:     user <user@spark.apache.org>
Date:   01/02/2016 07:03 pm
Subject:        Re: Guidelines for writing SPARK packages
Sent by:        marchoffolly@gmail.com



Hi Praveen,

The basic requirements for releasing a Spark package on
spark-packages.org are as follows:

1. The package content must be hosted by GitHub in a public repo under
the owner's account.
2. The repo name must match the package name.
3. The master branch of the repo must contain "README.md" and "LICENSE".

Per the doc on spark-packages.org site an example package that meets
those requirements can be found at
https://github.com/databricks/spark-avro. My own recently released
SAMBA package also meets these requirements:
https://github.com/onetapbeyond/lambda-spark-executor.

As you can see there is nothing in this list of requirements that
demands the implementation of specific interfaces. What you'll need to
implement will depend entirely on what you want to accomplish. If you
want to register a release for your package you will also need to push
the artifacts for your package to Maven central.

David


On Mon, Feb 1, 2016 at 7:03 AM, Praveen Devarao <praveendrl@in.ibm.com> 
wrote:
> Hi,
>
>         Is there any guidelines or specs to write a Spark package? I 
would
> like to implement a spark package and would like to know the way it 
needs to
> be structured (implement some interfaces etc) so that it can plug into 
Spark
> for extended functionality.
>
>         Could any one help me point to docs or links on the above?
>
> Thanking You
>
> Praveen Devarao



-- 
"All that is gold does not glitter, Not all those who wander are lost."

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org






Mime
View raw message