spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aakash Basu <aakash.spark....@gmail.com>
Subject Accessing a dataframe from another Singleton class (Python)
Date Mon, 13 Aug 2018 06:47:32 GMT
Hi,

I wanted to read a dataframe in one singleton class and and use that in
another singleton class.

Below is my code -

Class Singleton -

class Singleton(object):
  _instances = {}
  def __new__(class_, *args, **kwargs):
    if class_ not in class_._instances:
        class_._instances[class_] = super(Singleton,
class_).__new__(class_, *args, **kwargs)
    return class_._instances[class_]


Class A -


from experiment.Singleton import Singleton

class ReadData(Singleton):

data = None
test_field=None

def run(self):
    self.data = self.spark.read.option("header", "true").csv(data_path)

    self.test_field = 234


Class B -


from experiment.Singleton import Singleton

class Clustering(Singleton):

data = None
test_field=None

def run(self):
    read_class_object = ReadData()

    number = read_class_object.test_field

    print(number)
    X = read_class_object.data
    print('Showing the DATA READ at Clustering block - ', X)

    X.show()


from test_df_bb.classA import ReadData
from test_df_bb.classB import Clustering

obj1 = ReadData()
obj2 = Clustering()

obj1.init()
obj1.run()

obj2.init()
obj2.run()


In the second class, the normal pythonic values like string and integers
are working but the dataframe is coming as None.

Is there any ideal way to call another class and get a dataframe it read
and do operations on several classes? Please help!


Thanks,
Aakash.

Mime
View raw message