spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Erlandson <eerla...@redhat.com>
Subject Kubernetes Big-Data-SIG notes, September 5
Date Wed, 05 Sep 2018 21:26:22 GMT
Meta
At the weekly K8s Big Data SIG meeting today, we agreed to experiment with
publishing a brief summary of noteworthy Spark-related topics from the
weekly meeting to dev@spark, as a reference for interested members of the
Apache Spark community.

The format is a brief summary, including a link to the SIG minutes (we also
post a link to the meeting recording on the minutes when it becomes
available).

With that, here are the first SIG meeting notes:

Spark
Based on initial feedback on the remote shuffle service storage design
exploration <https://issues.apache.org/jira/browse/SPARK-25299>, further
work on design options is going to continue prior to attempting a POC
implementation. The group is consulting Facebook and Baidu for additional
input from their independent work in this area.

We reviewed the new fractional CPU
<https://issues.apache.org/jira/browse/SPARK-23285> support on the K8s
back-end (landing in Spark 2.4) and discussed options for configuring
fractional CPUs on the driver pod.

A consensus was reached to proceed with the current PR
<https://github.com/apache/spark/pull/22146> on apache/spark for the
upcoming user supplied pod template feature on the K8s back-end.

Link to meeting minutes
https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85
Vms4V2uTsGZvSp8MNIA

Mime
View raw message