spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Taotao.Li" <charles.up...@gmail.com>
Subject Re: Hi, guys, does anyone use Spark in finance market?
Date Thu, 01 Sep 2016 23:51:41 GMT
Hi, Adam, great thanks for your detailed reply, the three videos are
very referential for me. Actually, the App submitted to IBM Spark Contest
is a very small demo, I'll do much more work to enhance that model, and
recently we just started a new project which aims to building a platform
that makes it possible and easy for Chinese financial institutions to
analysis high-frequency market data, which will be ~30G one day.


great thanks again,

On Thu, Sep 1, 2016 at 10:25 PM, Adam Roberts <AROBERTS@uk.ibm.com> wrote:

> Hi, yes, there's definitely a market for Apache Spark and financial
> institutions, I can't provide specific details but to answer your survey:
> "yes" and "more than a few GB!"
>
> Here are a couple of examples showing Spark with financial data, full
> disclosure that I work for IBM, I'm sure there are lots more examples you
> can find too:
>
>    - https://www.youtube.com/watch?v=VWBNoIwGEjo shows how Spark can be
>    used with simple sentiment analysis to figure out correlations between real
>    world events and stock market changes. The Spark specific part is from 3 04
>    until the 8th minute
>    - https://www.youtube.com/watch?v=sDmWcuO5Rk8 is a similar example
>    where Spark is again used with sentiment analysis. One could also analyse
>    financial data to identify trends, I think a lot of the machine learning
>    APIs will be useful here e.g. logistic regression with many features could
>    be used to decide whether or not an investment is a good idea based on
>    training data (so we'd look at real outcomes from previous speculations)
>
>
> In both cases you can see Spark is a very important component for
> performing the calculations with financial data.
>
> I also know that Goldman Sachs mentioned they are interested in Spark, one
> talk is at https://www.youtube.com/watch?v=HWwAoTK2YrQ, so this is more
> evidence of the financial industries paying attention to big data and Spark.
>
> Regarding your app: I expected it to be similar to the first example where
> the signals you mention are real world events (e.g. the fed lowers interest
> rates or companies are rumoured to either be about to float or be
> acquired).
>
> At the 4 30 part I think you actually identify previous index values and
> extrapolate what they are likely to become using, so in theory your system
> would become more accurate over time although would going off indexes alone
> be sufficient (if indeed this is what you're doing).
>
> I think you'd want to combine this with real world speculation/news to
> figure out *why* the price is likely to change, how much by and in which
> direction.
>
> I agree that Apache Spark can be just the right tool for doing the heavy
> lifting required for analysis, computation and modelling of big data so
> looking forward to future Spark work in this area, and I wonder how we as
> Spark developers can make it easier/more powerful for Spark users to do so
>
>
>
>
> From:        "Taotao.Li" <charles.upboy@gmail.com>
> To:        user <user@spark.apache.org>
> Date:        30/08/2016 14:14
> Subject:        Hi, guys, does anyone use Spark in finance market?
> ------------------------------
>
>
>
>
> Hi, guys,
>
>      I'm a quant engineer in China, and I believe it's very promising when
> using Spark in the financial market. But I didn't find cases which combine
> spark and finance.
>
>     So here I wanna do a small survey:
>
>    - do you guys use Spark in financial market related project?
>    - if yes, how large data was fed in your spark application?
>
>
>      thanks a lot.
>
> *___________________**​*
> ​A little ad, I attended IBM Spark Hackathon, which is here :
> *http://apachespark.devpost.com/* <http://apachespark.devpost.com/> , and
> I submitted a small application, which will be used in my strategies, hope
> you guys and give me a vote and some suggestions on how to use spark in
> financial market, to discover some trade opportunity.
>
> here is my small app:
> *http://devpost.com/software/spark-in-finance-quantitative-investing*
> <http://devpost.com/software/spark-in-finance-quantitative-investing>
>
> thanks a lot.​
>
>
> --
> *___________________*
> Quant | Engineer | Boy
> *___________________*
> *blog*:    *http://litaotao.github.io*
> <http://litaotao.github.io?utm_source=spark_mail>
> *github*: *www.github.com/litaotao* <http://www.github.com/litaotao>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>



-- 
*___________________*
Quant | Engineer | Boy
*___________________*
*blog*:    http://litaotao.github.io
<http://litaotao.github.io?utm_source=spark_mail>
*github*: www.github.com/litaotao

Mime
View raw message