i would like to change some defaults in spark without overriding the user if she/he wishes to change them.

for example currently spark.blacklist.enabled is by default false, which makes sense for backwards compatibility.

i would like it to be by default true, but if the user provided --conf spark.spark.blacklist.enabled=false it should be set to false again.

i am aware that i can do:

val spark = SparkSession
  .config("spark.spark.blacklist.enabled", "true")

however if i understand the order in which settings are applied correctly, this will override any setting the user provided with --conf, which is undesired.

note that i do not have permission to change spark-defaults.conf because the spark install is also shared with some other applications for which i do not want to change the defaults.

any suggestions? thanks