spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (Jira)" <>
Subject [jira] [Updated] (SPARK-30244) Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Date Mon, 16 Mar 2020 22:52:08 GMT


Dongjoon Hyun updated SPARK-30244:
    Affects Version/s:     (was: 3.0.0)

> Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
> ---------------------------------------------------------------------------
>                 Key: SPARK-30244
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.1.0
>            Reporter: Jackson Westeen
>            Priority: Major
> Many of the methods in `ExternalCatalogWithListener` support emitting pre/post events
to the ListenerBus when delegating calls to the underlying ExternalCatalog implementation. 
> Is there a reason the "Partition" series of methods don't support this feature?
> []
> We're looking to hook into post "ALTER TABLE ADD PARTITION" calls, so we can publish
a message to Kafka for example, whenever new partitions are added to the metastore.
> It appears there's already event hooks for createDatabase / createTable.. but not createPartition.
Would you consider adding hooks here? Or is there a better way to hook into to successful
"ALTER TABLE ADD PARTITION" calls and subscribe to changes in partition status?
> Thank you!

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message