spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Gaido <>
Subject Re: Documentation on org.apache.spark.sql.functions backend.
Date Mon, 16 Sep 2019 08:07:07 GMT
Hi Vipul,

a function is never turned in a logical plan. A function is turned into an
Expression. And an Expression can be part of many Logical or Physical Plans.
Hope this helps.


Il giorno lun 16 set 2019 alle ore 08:27 Vipul Rajan <>
ha scritto:

> I am trying to create a function that reads data from Kafka, communicates
> with confluent schema registry and decodes avro data with evolving schemas.
> I am trying to not create hack-ish patches and to write proper code that I
> could maybe even create pull requests for. looking at the code I have been
> able to figure out a few things regarding how expressions are generated and
> how they help to accomplish what a function does, but there is still a ton
> I just cannot wrap my head around.
> I am unable to find any documentation that gets into such nitty gritties
> of Spark. *I am writing in hopes to find some help. Do you have any
> documentation that explains how a function
> (org.apache.spark.sql.function._) is turned into a logical plan?*

View raw message