spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Zhuge <>
Subject Re: SPIP: Catalog API for view metadata
Date Tue, 18 Aug 2020 16:46:21 GMT
> > AFAIK view schema is only used by DESCRIBE.
> Correction: Spark adds a new Project at the top of the parsed plan from
> view, based on the stored schema, to make sure the view schema doesn't
> change.

Thanks Wenchen! I thought I forgot something :) Yes it is the validation
done in *checkAnalysis*:

          // If the view output doesn't have the same number of columns
neither with the child
          // output, nor with the query column names, throw an
          // If the view's child output can't up cast to the view output,
          // throw an AnalysisException, too.

The view output comes from the schema:

      val child = View(
        desc = metadata,
        output = metadata.schema.toAttributes,
        child = parser.parsePlan(viewText))

So it is a validation (here) or cache (in DESCRIBE) nice to have but not
"required" or "should be frozen". Thanks Ryan and Burak for pointing that
out in SPIP. I will add a new paragraph accordingly.

View raw message