I need to extract a value from a PySpark structured streaming Dataframe to a string variable to check something.
I tried this code.
agentName = kinesisDF.select(kinesisDF.agentName.getItem(0).alias("agentName")).collect()
This works on a non-streaming Dataframe only. In a streaming Dataframe, collect is not supported.
Any workaround for this?