spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasad Bhalerao <prasadbhalerao1...@gmail.com>
Subject Re: reporting use case
Date Fri, 05 Apr 2019 01:19:53 GMT
Hi,

I am new to spark and no SQL databases.

So Please correct me if I am wrong.

Since I will be accessing multiple columns (almost 20-30 columns) of a row,
I will have to go with rowbased db instead column based right!
May be I can use Avro in this case. Does spark go well with Avroro? I will
do my research on this. But please let me know your opinion on this.

Thanks,
Prasad

On Fri 5 Apr, 2019, 1:09 AM Teemu Heikkilä <teemu@emblica.fi wrote:

> So basically you could have base dump/snapshot of the full database - or
> all the required data stored into HDFS or similar system as partitioned
> files (ie. orc/parquet)
>
> Then you use the change stream after the dump and join it on the snapshot
> - similarly than what your database is doing.
> After that you can build the aggregates and reports from that table.
>
> - T
>
> On 4 Apr 2019, at 22.35, Prasad Bhalerao <prasadbhalerao1983@gmail.com>
> wrote:
>
> I did not understand this "update actual snapshots ie. by joining the
> data".
>
>
> There is another microservice which updates these Oracle tables. I can
> have this micro service to send the update data feed on Kafka topics.
>
> Thanks,
> Prasad
>
> On Fri 5 Apr, 2019, 12:57 AM Teemu Heikkilä <teemu@emblica.fi wrote:
>
>> Based on your answers, I would consider using the update stream to update
>> actual snapshots ie. by joining the data
>>
>> Ofcourse now it depends on how the update stream has been implemented how
>> to get the data in spark.
>>
>> Could you tell little bit more about that?
>> - Teemu
>>
>> On 4 Apr 2019, at 22.23, Prasad Bhalerao <prasadbhalerao1983@gmail.com>
>> wrote:
>>
>> Hi ,
>>
>> I can create a view on these tables but the thing is I am going to need
>> almost every column from these tables and I have faced issues with oracle
>> views on such a large tables which involves joins. Some how oracle used to
>> choose not so correct execution plan.
>>
>> Can you please tell me how creating a views will help in this scenario?
>>
>> Can you please tell if I am thinking in right direction?
>>
>> I have two challenges
>> 1) First to load 2-4 TB of data in spark very quickly.
>> 2) And then keep this data updated in spark whenever DB updates are done.
>>
>> Thanks,
>> Prasad
>>
>> On Fri, Apr 5, 2019 at 12:35 AM Jason Nerothin <jasonnerothin@gmail.com>
>> wrote:
>>
>>> Hi Prasad,
>>>
>>> Could you create an Oracle-side view that captures only the relevant
>>> records and the use Spark JDBC connector to load the view into Spark?
>>>
>>> On Thu, Apr 4, 2019 at 1:48 PM Prasad Bhalerao <
>>> prasadbhalerao1983@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am exploring spark for my Reporting application.
>>>> My use case is as follows...
>>>> I have 4-5 oracle tables which contains more than 1.5 billion rows.
>>>> These tables are updated very frequently every day. I don't have choice to
>>>> change database technology. So this data is going to remain in Oracle only.
>>>> To generate 1 report, on an average 15 - 50 million rows has to be
>>>> fetched from oracle tables. These rows contains some blob columns. Most of
>>>> the time is spent in fetching these many rows from db over the network.
>>>> Data processing is not that complex. Currently these report takes around
>>>> 3-8 hours to complete. I trying to speed up this report generation process.
>>>>
>>>> Can use spark as a caching layer in this case to avoid fetching data
>>>> from oracle over the network every time? I am thinking to submit a spark
>>>> job for each report request and use spark SQL to fetch the data and then
>>>> process it and write to a file? I trying to use kind of data locality in
>>>> this case.
>>>>
>>>> Whenever a data is updated in oracle tables can I refresh the data in
>>>> spark storage? I can get the update feed using messaging technology.
>>>>
>>>> Can some one from community help me with this?
>>>> Suggestions are welcome.
>>>>
>>>>
>>>> Thanks,
>>>> Prasad
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Prasad
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Jason
>>>
>>
>>
>

Mime
View raw message