I have an external catalog that has additional information on my Parquet files that I want to match up with the parsed filters from the plan to prune the list of files included in the scan. I’m looking at doing this using the Spark 2.2.0 SparkSession extensions similar to the built in partition pruning:
and this other project that is along the lines of what I want:
but isn’t caught up to 2.2.0, but I’m struggling to understand what type of extension I would use to do something like the above:
and if this is the appropriate strategy for this.
Are there any examples out there for using the new extension hooks to alter the files included in the plan?