spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Spark ML Pipeline inaccessible types
Date Wed, 25 Mar 2015 11:04:16 GMT
NoSuchMethodError in general means that your runtime and compile-time
environments are different. I think you need to first make sure you
don't have mismatching versions of Spark.

On Wed, Mar 25, 2015 at 11:00 AM,  <> wrote:
> Hi,
> I have started implementing a machine learning pipeline using Spark 1.3.0
> and the new pipelining API and DataFrames. I got to a point where I have my
> training data set prepared using a sequence of Transformers, but I am
> struggling to actually train a model and use it for predictions.
> I am getting a java.lang.NoSuchMethodException:
> exception thrown at checkInputColumn method in Params trait when using a
> Predictor (LinearRegression in my case, but that should not matter). This
> looks like a bug - the exception is thrown when executing getParam(colName)
> when the require(actualDataType.equals(datatype), ...) requirement is not
> met so the expected requirement failed exception is not thrown and is hidden
> by the unexpected NoSuchMethodException instead. I can raise a bug if this
> really is an issue and I am not using something incorrectly.
> The problem I am facing however is that the Predictor expects features to
> have VectorUDT type as defined in Predictor class (protected def
> featuresDataType: DataType = new VectorUDT). But since this type is
> private[spark] my Transformer can not prepare features with this type which
> then correctly results in the exception above when I use a different type.
> Is there a way to define a custom Pipeline that would be able to use the
> existing Predictors without having to bypass the access modifiers or
> reimplement something or is the pipelining API not yet expected to be used
> in this way?
> Thanks,
> Martin

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message