systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From arijit chakraborty <ak...@hotmail.com>
Subject Re: Install - Configure Jupyter Notebook
Date Sun, 02 Jul 2017 11:22:07 GMT
Hi Gustavo,


You can put that pyspark details in the jupyter console itself.


import os
import sys
import pandas as pd
import numpy as np

spark_path = "C:\spark"
os.environ['SPARK_HOME'] = spark_path
os.environ['HADOOP_HOME'] = spark_path

sys.path.append(spark_path + "/bin")
sys.path.append(spark_path + "/python")
sys.path.append(spark_path + "/python/pyspark/")
sys.path.append(spark_path + "/python/lib")
sys.path.append(spark_path + "/python/lib/pyspark.zip")
sys.path.append(spark_path + "/python/lib/py4j-0.10.4-src.zip")

from pyspark import SparkContext
from pyspark import SparkConf

sc = SparkContext("local[*]", "test")


# SystemML Specifications:


from pyspark.sql import SQLContext
import systemml as sml
sqlCtx = SQLContext(sc)
ml = sml.MLContext(sc)


But this is not a very good way of doing it. I did it as I'm using windows and it's easier
to do it like that.


Regards,

Arijit

________________________________
From: Gustavo Frederico <gustavo.frederico@thinkwrap.com>
Sent: Sunday, July 2, 2017 10:16:03 AM
To: dev@systemml.apache.org
Subject: Install - Configure Jupyter Notebook


A basic question: step 3 in https://systemml.apache.org/install-systemml.html <https://systemml.apache.org/install-systemml.html>
 for “Configure Jupyter Notebook” has
# Start Jupyter Notebook Server
PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS="notebook" pyspark --master local[*]
--conf "spark.driver.memory=12g" --conf spark.driver.maxResultSize=0 --conf spark.akka.frameSize=128
--conf spark.default.parallelism=100
Where does that go? There are no details in this step…

Thanks

Gustavo

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message