Creating a column independently with value output from an API

Asked Sep 13 '19 at 19:43

Active Sep 14 '19 at 04:06

Viewed 72 times

I need to create a Spark column in Java with the following requirements: - Column should be created independently. i.e. should be created without any knowledge on other columns in the dataset - Should have a custom unique value per row returned from an API, e.g: UUID.randomUUID().toString() or MYCLASS.customMethod();

I have tried the following:

1) Using a UDF - it works but this is deterministic (Spark Dataframe Random UUID changes after every transformation/action)

2) Used expr("uuid()"), but this uses the internal implementation and I want to use my own API here.

Any leads on how to create a column independently with the value of an API is appreciated (in Java). Considering the upstream process is deterministic. Additionally, I am looking at how the sql uuid() was implemented.

Thoughts for now: can I create a custom Expression in Java and pass to the Column constructor.

edited Sep 14 '19 at 04:06

DennisLi

3,915
6
30
66

asked Sep 13 '19 at 19:43

Malaka

1

Check `MontotonicallyIncreasingID` or `ZipWithUniqueId` – SMaZ Sep 13 '19 at 21:38
Did you try https://stackoverflow.com/a/42367629/11900243 ? – Aleksei Sep 14 '19 at 07:51
@sMaz MontotonicallyIncreasingID and ZipWithUniqueId provide Ids but I want to provide the Java implementation for the value generation. – Malaka Sep 19 '19 at 07:04
@Aleksei rand returns a unique value, but I want to provide the Java implementation for the value generation. – Malaka Sep 19 '19 at 07:05

Creating a column independently with value output from an API

0 Answers0