spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adline Dsilva <>
Subject Resizing Image with Scrimage in Spark
Date Mon, 17 Oct 2016 07:03:57 GMT
Hi All,

   I have a Hive Table which contains around 500 million photos(Profile picture of Users)
stored as hex string and total size of the table is 5TB. I'm trying to make a solution where
images can be retrieved in real-time.

Current Solution,  Resize the images, index it along the user profile to solr. For Resizing,
Im using a scala library called scrimage<>

While running the udf function im getting below error.
Serialization stack:
    - object not serializable (class: com.sksamuel.scrimage.Image, value: Image [width=767,
height=1024, type=2])
    - field (class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC, name: imgR, type: class com.sksamuel.scrimage.Image)

Can anyone suggest method to overcome the above error.



This e-mail (including any attachments) is for the addressee(s) only and may be confidential,
especially as regards personal data. If you are not the intended recipient, please note that
any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited.
If you have received this email in error, please notify the sender immediately and delete
the original message (including any attachments).

MIMOS Berhad is a research and development institution under the purview of the Malaysian
Ministry of Science, Technology and Innovation. Opinions, conclusions and other information
in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries
shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries
and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability
arising from or in connection with computer viruses and/or corrupted e-mails is excluded to
the fullest extent permitted by law.

View raw message