I was a little sloppy when I created the sample output. Its missing a few pairs

Assume for a given row I have [a, b, c] I want to create something like the cartesian join

From: Andrew Davidson <Andy@SantaCruzIntegration.com>
Date: Friday, March 30, 2018 at 5:54 PM
To: "user @spark" <user@spark.apache.org>
Subject: how to create all possible combinations from an array? how to join and explode row array?

I have a dataframe and execute  df.groupBy(“xyzy”).agg( collect_list(“abc”)

This produces a column of type array. Now for each row I want to create a multiple pairs/tuples from the array so that I can create a contingency table.  Any idea how I can transform my data so that call crosstab() ? The join transformation operate on the entire dataframe. I need something at the row array level?

Bellow is some sample python and describes what I would like my results to be?

Kind regards


c1 = ["john", "bill", "sam"]
c2 = [['red', 'blue', 'red'], ['blue', 'red'], ['green']]
p = pd.DataFrame({"a":c1, "b":c2})

df = sqlContext.createDataFrame(p)

 |-- a: string (nullable = true)
 |-- b: array (nullable = true)
 |    |-- element: string (containsNull = true)

|   a|               b|
|john|[red, blue, red]|
|bill   |     [blue, red]|
| sam|         [green]|

The output I am trying to create is. I could live with a crossJoin (cartesian join) and add my own filtering if it makes the problem easier?

|  x1|    x2|
red  | blue
red  | red
blue | red