spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shyam P <shyamabigd...@gmail.com>
Subject Re: Is there any spark API function to handle a group of companies at once in this scenario?
Date Sat, 13 Apr 2019 00:54:42 GMT
Hi Mich,
Sorry sorry for late reply.
Yes this flow is near except HBase and Flume. Why do we need flume when we
use spark streaming?
 You did not address my base question yet.

Regards
Shyam


On Tue, 9 Apr 2019, 22:08 Mich Talebzadeh, <mich.talebzadeh@gmail.com>
wrote:

> Fine, how do you store these Kafka tropics? Are they loaded into HDFS via
> Kafka --> Flume --> HDFS?
>
> Look at this simple diagram below. Replace MongoDB with Cassandra. Is this
> what you are trying to do from an architecture point of view?
>
>
> [image: image.png]
>
> HTH,
>
> Nuch
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 8 Apr 2019 at 08:19, Shyam P <shyamabigdata@gmail.com> wrote:
>
>> Hi Mich,
>>  thanks for your prompt reply.
>> I get few company financial data like profits and etc results .
>> I would get this company data through Kafka topics which is fed by an
>> rest service.
>> I am thinking of using spark-structured streaming.
>> Put them back in HIVE/C*.
>>
>> Regards,
>> Shyam
>>
>>
>>
>> On Sun, Apr 7, 2019 at 2:04 PM Mich Talebzadeh <mich.talebzadeh@gmail.com>
>> wrote:
>>
>>> Are these ticker prices for these companies like share value etc?
>>>
>>> How do you get this company data in Spark? Are you using Spark streaming
>>> to get the prices, then work out the stats (AVG, STDDEV etc) and put them
>>> back into DB?
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Fri, 5 Apr 2019 at 10:51, Shyam P <shyamabigdata@gmail.com> wrote:
>>>
>>>> Hi ,
>>>> In my scenario I have few companies , for which I need to calculate few
>>>> stats like avg I need to be stored in Cassandra , for next set of records
I
>>>> need to get previously calculated and over it i need to calculate
>>>> accumulated results ( i.e preset set of data + previously stored stats) and
>>>> stored it back to Cassandra.
>>>>
>>>> what function/API of spark be used while calculating the above for a
>>>> group of companies?
>>>>
>>>>
>>>> Regards,
>>>> Shyam
>>>>
>>>

Mime
View raw message