spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Is there any spark API function to handle a group of companies at once in this scenario?
Date Tue, 09 Apr 2019 16:38:44 GMT
Fine, how do you store these Kafka tropics? Are they loaded into HDFS via
Kafka --> Flume --> HDFS?

Look at this simple diagram below. Replace MongoDB with Cassandra. Is this
what you are trying to do from an architecture point of view?


[image: image.png]

HTH,

Nuch


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Mon, 8 Apr 2019 at 08:19, Shyam P <shyamabigdata@gmail.com> wrote:

> Hi Mich,
>  thanks for your prompt reply.
> I get few company financial data like profits and etc results .
> I would get this company data through Kafka topics which is fed by an rest
> service.
> I am thinking of using spark-structured streaming.
> Put them back in HIVE/C*.
>
> Regards,
> Shyam
>
>
>
> On Sun, Apr 7, 2019 at 2:04 PM Mich Talebzadeh <mich.talebzadeh@gmail.com>
> wrote:
>
>> Are these ticker prices for these companies like share value etc?
>>
>> How do you get this company data in Spark? Are you using Spark streaming
>> to get the prices, then work out the stats (AVG, STDDEV etc) and put them
>> back into DB?
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Fri, 5 Apr 2019 at 10:51, Shyam P <shyamabigdata@gmail.com> wrote:
>>
>>> Hi ,
>>> In my scenario I have few companies , for which I need to calculate few
>>> stats like avg I need to be stored in Cassandra , for next set of records I
>>> need to get previously calculated and over it i need to calculate
>>> accumulated results ( i.e preset set of data + previously stored stats) and
>>> stored it back to Cassandra.
>>>
>>> what function/API of spark be used while calculating the above for a
>>> group of companies?
>>>
>>>
>>> Regards,
>>> Shyam
>>>
>>

Mime
View raw message