spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Roberts <>
Subject Re: Hi, guys, does anyone use Spark in finance market?
Date Thu, 01 Sep 2016 14:25:56 GMT
Hi, yes, there's definitely a market for Apache Spark and financial 
institutions, I can't provide specific details but to answer your survey: 
"yes" and "more than a few GB!"

Here are a couple of examples showing Spark with financial data, full 
disclosure that I work for IBM, I'm sure there are lots more examples you 
can find too: shows how Spark can be used 
with simple sentiment analysis to figure out correlations between real 
world events and stock market changes. The Spark specific part is from 3 
04 until the 8th minute is a similar example where 
Spark is again used with sentiment analysis. One could also analyse 
financial data to identify trends, I think a lot of the machine learning 
APIs will be useful here e.g. logistic regression with many features could 
be used to decide whether or not an investment is a good idea based on 
training data (so we'd look at real outcomes from previous speculations)

In both cases you can see Spark is a very important component for 
performing the calculations with financial data.

I also know that Goldman Sachs mentioned they are interested in Spark, one 
talk is at, so this is more 
evidence of the financial industries paying attention to big data and 

Regarding your app: I expected it to be similar to the first example where 
the signals you mention are real world events (e.g. the fed lowers 
interest rates or companies are rumoured to either be about to float or be 

At the 4 30 part I think you actually identify previous index values and 
extrapolate what they are likely to become using, so in theory your system 
would become more accurate over time although would going off indexes 
alone be sufficient (if indeed this is what you're doing). 

I think you'd want to combine this with real world speculation/news to 
figure out *why* the price is likely to change, how much by and in which 

I agree that Apache Spark can be just the right tool for doing the heavy 
lifting required for analysis, computation and modelling of big data so 
looking forward to future Spark work in this area, and I wonder how we as 
Spark developers can make it easier/more powerful for Spark users to do so

From:   "Taotao.Li" <>
To:     user <>
Date:   30/08/2016 14:14
Subject:        Hi, guys, does anyone use Spark in finance market?

Hi, guys,

     I'm a quant engineer in China, and I believe it's very promising when 
using Spark in the financial market. But I didn't find cases which combine 
spark and finance.

    So here I wanna do a small survey: 

do you guys use Spark in financial market related project?
if yes, how large data was fed in your spark application?

     thanks a lot.

​A little ad, I attended IBM Spark Hackathon, which is here : , and I submitted a small application, 
which will be used in my strategies, hope you guys and give me a vote and 
some suggestions on how to use spark in financial market, to discover some 
trade opportunity.

here is my small app:

thanks a lot.​

Quant | Engineer | Boy

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

View raw message