spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Taotao.Li" <>
Subject Re: Hi, guys, does anyone use Spark in finance market?
Date Thu, 01 Sep 2016 23:51:41 GMT
Hi, Adam, great thanks for your detailed reply, the three videos are
very referential for me. Actually, the App submitted to IBM Spark Contest
is a very small demo, I'll do much more work to enhance that model, and
recently we just started a new project which aims to building a platform
that makes it possible and easy for Chinese financial institutions to
analysis high-frequency market data, which will be ~30G one day.

great thanks again,

On Thu, Sep 1, 2016 at 10:25 PM, Adam Roberts <> wrote:

> Hi, yes, there's definitely a market for Apache Spark and financial
> institutions, I can't provide specific details but to answer your survey:
> "yes" and "more than a few GB!"
> Here are a couple of examples showing Spark with financial data, full
> disclosure that I work for IBM, I'm sure there are lots more examples you
> can find too:
>    - shows how Spark can be
>    used with simple sentiment analysis to figure out correlations between real
>    world events and stock market changes. The Spark specific part is from 3 04
>    until the 8th minute
>    - is a similar example
>    where Spark is again used with sentiment analysis. One could also analyse
>    financial data to identify trends, I think a lot of the machine learning
>    APIs will be useful here e.g. logistic regression with many features could
>    be used to decide whether or not an investment is a good idea based on
>    training data (so we'd look at real outcomes from previous speculations)
> In both cases you can see Spark is a very important component for
> performing the calculations with financial data.
> I also know that Goldman Sachs mentioned they are interested in Spark, one
> talk is at, so this is more
> evidence of the financial industries paying attention to big data and Spark.
> Regarding your app: I expected it to be similar to the first example where
> the signals you mention are real world events (e.g. the fed lowers interest
> rates or companies are rumoured to either be about to float or be
> acquired).
> At the 4 30 part I think you actually identify previous index values and
> extrapolate what they are likely to become using, so in theory your system
> would become more accurate over time although would going off indexes alone
> be sufficient (if indeed this is what you're doing).
> I think you'd want to combine this with real world speculation/news to
> figure out *why* the price is likely to change, how much by and in which
> direction.
> I agree that Apache Spark can be just the right tool for doing the heavy
> lifting required for analysis, computation and modelling of big data so
> looking forward to future Spark work in this area, and I wonder how we as
> Spark developers can make it easier/more powerful for Spark users to do so
> From:        "Taotao.Li" <>
> To:        user <>
> Date:        30/08/2016 14:14
> Subject:        Hi, guys, does anyone use Spark in finance market?
> ------------------------------
> Hi, guys,
>      I'm a quant engineer in China, and I believe it's very promising when
> using Spark in the financial market. But I didn't find cases which combine
> spark and finance.
>     So here I wanna do a small survey:
>    - do you guys use Spark in financial market related project?
>    - if yes, how large data was fed in your spark application?
>      thanks a lot.
> *___________________**​*
> ​A little ad, I attended IBM Spark Hackathon, which is here :
> ** <> , and
> I submitted a small application, which will be used in my strategies, hope
> you guys and give me a vote and some suggestions on how to use spark in
> financial market, to discover some trade opportunity.
> here is my small app:
> **
> <>
> thanks a lot.​
> --
> *___________________*
> Quant | Engineer | Boy
> *___________________*
> *blog*:    **
> <>
> *github*: ** <>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Quant | Engineer | Boy

View raw message