spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Erlandson <>
Subject Kubernetes Big-Data-SIG notes, September 5
Date Wed, 05 Sep 2018 21:26:22 GMT
At the weekly K8s Big Data SIG meeting today, we agreed to experiment with
publishing a brief summary of noteworthy Spark-related topics from the
weekly meeting to dev@spark, as a reference for interested members of the
Apache Spark community.

The format is a brief summary, including a link to the SIG minutes (we also
post a link to the meeting recording on the minutes when it becomes

With that, here are the first SIG meeting notes:

Based on initial feedback on the remote shuffle service storage design
exploration <>, further
work on design options is going to continue prior to attempting a POC
implementation. The group is consulting Facebook and Baidu for additional
input from their independent work in this area.

We reviewed the new fractional CPU
<> support on the K8s
back-end (landing in Spark 2.4) and discussed options for configuring
fractional CPUs on the driver pod.

A consensus was reached to proceed with the current PR
<> on apache/spark for the
upcoming user supplied pod template feature on the K8s back-end.

Link to meeting minutes

View raw message