spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: A question on extrapolation of a nonlinear curve fit beyond x value
Date Tue, 05 Jan 2021 15:38:37 GMT
Thanks again

Just to clarify, I want to see the average price for year 2021, 2022 etc
based on the best fit. So naively if someone asked a question what the
average price will be in 2022, I should be able to make some predictions.

I can of course crudely use pen and pencil like shown in the attached
figure, but I was wondering if this is possible with anything that
matplotlib offers?



[image: Capture123.PNG]



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Tue, 5 Jan 2021 at 15:22, Sean Owen <srowen@gmail.com> wrote:

> You will need to use matplotlib on the driver to plot in any event. If
> this is a single extrapolation, over 11 data points, you can just use Spark
> to do the aggregation, call .toPandas, and do whatever you want in the
> Python ecosystem to fit and plot that result.
>
> On Tue, Jan 5, 2021 at 9:18 AM Mich Talebzadeh <mich.talebzadeh@gmail.com>
> wrote:
>
>> thanks Sean.
>>
>> This is the gist of the case
>>
>> <https://stackoverflow.com/posts/65570917/timeline>
>>
>> I have data points for x-axis from 2010 till 2020 and values for y axis.
>> I am using PySpark, pandas and matplotlib. Data is read into PySpark from
>> the underlying database and a pandas Data Frame is built on it. Data is
>> aggregated over each year. However, the underlying prices are provided on a
>> monthly basis in CSV file which has been loaded into a Hive table
>>
>> summary_df = spark.sql(f"""SELECT cast(Year as int) as year,
>> AVGFlatPricePerYear, AVGTerracedPricePerYear, AVGSemiDetachedPricePerYear,
>> AVGDetachedPricePerYear FROM {v.DSDB}.yearlyhouseprices""")
>>
>> df_10 = summary_df.filter(col("year").between(f'{start_date}',
>> f'{end_date}'))
>>
>> p_dfm = df_10.toPandas()  # converting spark DF to Pandas DF
>>
>>
>> for i in range(n):
>>
>>   if p_dfm.columns[i] != 'year':   # year is x axis in integer
>>
>>     vcolumn = p_dfm.columns[i]
>>
>>      print(vcolumn)
>>
>>      params = model.guess(p_dfm[vcolumn], x = p_dfm['year'])
>>
>>      result = model.fit(p_dfm[vcolumn], params, x = p_dfm['year'])
>>
>>      result.plot_fit()
>>
>>      if vcolumn == "AVGFlatPricePerYear":
>>
>>          plt.xlabel("Year", fontdict=v.font)
>>
>>          plt.ylabel("Flat house prices in millions/GBP", fontdict=v.font)
>>
>>          plt.title(f"""Flat price fluctuations in {regionname} for the
>> past 10                 years """,  fontdict=v.font)
>>
>>          plt.text(0.35,
>>
>>             0.45,
>>
>>             "Best-fit based on Non-Linear Lorentzian Model",
>>
>>             transform=plt.gca().transAxes,
>>
>>             color="grey",
>>
>>             fontsize=10
>>
>>          )
>>
>>          print(result.fit_report())
>>
>>          plt.xlim(left=2009)
>>
>>          plt.xlim(right=2022)
>>
>>          plt.show()
>>
>>          plt.close()
>>
>> ```
>>
>> So far so good. I get a best fit plot as shown using Lorentzian model
>>
>> Also I have model fit data
>>
>> [[Model]]
>>
>>     Model(lorentzian)
>>
>> [[Fit Statistics]]
>>
>>     # fitting method   = leastsq
>>
>>     # function evals   = 25
>>
>>     # data points      = 11
>>
>>     # variables        = 3
>>
>>     chi-square         = 8.4155e+09
>>
>>     reduced chi-square = 1.0519e+09
>>
>>     Akaike info crit   = 231.009958
>>
>>     Bayesian info crit = 232.203644
>>
>> [[Variables]]
>>
>>     amplitude:  31107480.0 +/- 1471033.33 (4.73%) (init = 6106104)
>>
>>     center:     2016.75722 +/- 0.18632315 (0.01%) (init = 2016.5)
>>
>>     sigma:      8.37428353 +/- 0.45979189 (5.49%) (init = 3.5)
>>
>>     fwhm:       16.7485671 +/- 0.91958379 (5.49%) == '2.0000000*sigma'
>>
>>     height:     1182407.88 +/- 15681.8211 (1.33%) ==
>> '0.3183099*amplitude/max(2.220446049250313e-16, sigma)'
>>
>> [[Correlations]] (unreported correlations are < 0.100)
>>
>>     C(amplitude, sigma)  =  0.977
>>
>>     C(amplitude, center) =  0.644
>>
>>     C(center, sigma)     =  0.603
>>
>>
>> Now I need to predict the prices for years 2021-2022 based on this fit.
>> Is there any way I can use some plt functions to provide extrapolated
>> values for 2021 and beyond?
>>
>>
>> Thanks
>>
>>
>>
>>
>>
>> On Tue, 5 Jan 2021 at 14:43, Sean Owen <srowen@gmail.com> wrote:
>>
>>> If your data set is 11 points, surely this is not a distributed problem?
>>> or are you asking how to build tens of thousands of those projections in
>>> parallel?
>>>
>>> On Tue, Jan 5, 2021 at 6:04 AM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am not sure Spark forum is the correct avenue for this question.
>>>>
>>>> I am using PySpark with matplotlib to  get the best fit for data using
>>>> the Lorentzian Model. This curve uses 2010-2020 data points (11 on x-axis).
>>>> I need to predict predict the prices for years 2021-2025 based on this
>>>> fit. So not sure if someone can advise me? If Ok, then I can post the
>>>> details
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>

Mime
View raw message