spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Taotao.Li" <charles.up...@gmail.com>
Subject Re: Hi, guys, does anyone use Spark in finance market?
Date Thu, 01 Sep 2016 23:54:21 GMT
Hi, Mich, thanks for your detailed reply, that system seems interesting,
I'll take a deep look on it, great thanks again.

On Fri, Sep 2, 2016 at 6:54 AM, Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:

> I have worked for a long time in the Financial sector and Spark plus Spark
> Streaming is used for both Market risk and Credit risk calculations.
>
> I am currently working on a design pattern of mine to use Lambda
> Architecture for Financial risk. Big data techniques and applications are
> providing data-intensive industries such as banks a solution to extract
> latent information from the vast volume of data accumulated from all the
> lines of enterprise business. Given the volume of data and the available
> resources, the banks have now begun to focus on the benefits that big data
> techniques can bring in streamlining current practices, as well as
> designing new solutions. A high level design of mine is attached.
>
>
>
> ​
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 1 September 2016 at 15:25, Adam Roberts <AROBERTS@uk.ibm.com> wrote:
>
>> Hi, yes, there's definitely a market for Apache Spark and financial
>> institutions, I can't provide specific details but to answer your survey:
>> "yes" and "more than a few GB!"
>>
>> Here are a couple of examples showing Spark with financial data, full
>> disclosure that I work for IBM, I'm sure there are lots more examples you
>> can find too:
>>
>>    - https://www.youtube.com/watch?v=VWBNoIwGEjo shows how Spark can be
>>    used with simple sentiment analysis to figure out correlations between real
>>    world events and stock market changes. The Spark specific part is from 3 04
>>    until the 8th minute
>>    - https://www.youtube.com/watch?v=sDmWcuO5Rk8 is a similar example
>>    where Spark is again used with sentiment analysis. One could also analyse
>>    financial data to identify trends, I think a lot of the machine learning
>>    APIs will be useful here e.g. logistic regression with many features could
>>    be used to decide whether or not an investment is a good idea based on
>>    training data (so we'd look at real outcomes from previous speculations)
>>
>>
>> In both cases you can see Spark is a very important component for
>> performing the calculations with financial data.
>>
>> I also know that Goldman Sachs mentioned they are interested in Spark,
>> one talk is at https://www.youtube.com/watch?v=HWwAoTK2YrQ, so this is
>> more evidence of the financial industries paying attention to big data and
>> Spark.
>>
>> Regarding your app: I expected it to be similar to the first example
>> where the signals you mention are real world events (e.g. the fed lowers
>> interest rates or companies are rumoured to either be about to float or be
>> acquired).
>>
>> At the 4 30 part I think you actually identify previous index values and
>> extrapolate what they are likely to become using, so in theory your system
>> would become more accurate over time although would going off indexes alone
>> be sufficient (if indeed this is what you're doing).
>>
>> I think you'd want to combine this with real world speculation/news to
>> figure out *why* the price is likely to change, how much by and in which
>> direction.
>>
>> I agree that Apache Spark can be just the right tool for doing the heavy
>> lifting required for analysis, computation and modelling of big data so
>> looking forward to future Spark work in this area, and I wonder how we as
>> Spark developers can make it easier/more powerful for Spark users to do so
>>
>>
>>
>>
>> From:        "Taotao.Li" <charles.upboy@gmail.com>
>> To:        user <user@spark.apache.org>
>> Date:        30/08/2016 14:14
>> Subject:        Hi, guys, does anyone use Spark in finance market?
>> ------------------------------
>>
>>
>>
>>
>> Hi, guys,
>>
>>      I'm a quant engineer in China, and I believe it's very promising
>> when using Spark in the financial market. But I didn't find cases which
>> combine spark and finance.
>>
>>     So here I wanna do a small survey:
>>
>>    - do you guys use Spark in financial market related project?
>>    - if yes, how large data was fed in your spark application?
>>
>>
>>      thanks a lot.
>>
>> *___________________**​*
>> ​A little ad, I attended IBM Spark Hackathon, which is here :
>> *http://apachespark.devpost.com/* <http://apachespark.devpost.com/> ,
>> and I submitted a small application, which will be used in my strategies,
>> hope you guys and give me a vote and some suggestions on how to use spark
>> in financial market, to discover some trade opportunity.
>>
>> here is my small app:
>> *http://devpost.com/software/spark-in-finance-quantitative-investing*
>> <http://devpost.com/software/spark-in-finance-quantitative-investing>
>>
>> thanks a lot.​
>>
>>
>> --
>> *___________________*
>> Quant | Engineer | Boy
>> *___________________*
>> *blog*:    *http://litaotao.github.io*
>> <http://litaotao.github.io?utm_source=spark_mail>
>> *github*: *www.github.com/litaotao* <http://www.github.com/litaotao>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>
>


-- 
*___________________*
Quant | Engineer | Boy
*___________________*
*blog*:    http://litaotao.github.io
<http://litaotao.github.io?utm_source=spark_mail>
*github*: www.github.com/litaotao

Mime
View raw message