Spark Streaming with Scala: is there a way which i can postpone dealing with data to next batch

Asked Jul 07 '16 at 13:03

Active Jul 07 '16 at 13:03

Viewed 33 times

I am working with Spark-Streaming + Scala, and i have the following scenario:

I need to create a Funnel (which is defined by page-views and clicks). A click must be connected to a page-view occurred before the click.

The problem might occur, when i get the click event in the batch before, the batch which i get the page-view event.

Is there a way which i can postpone the click event to the next event, and try to deal with it then ? Also a TTL is needed in such case, so we can throw the event after several batches.

thanks, Eran.

asked Jul 07 '16 at 13:03

EranM

1

Use stateful transformations like [`mapWithState`](http://stackoverflow.com/q/37550054/1560062) or `updateStateByKey`? – zero323 Jul 07 '16 at 14:51
i will try it. thanks ! – EranM Jul 10 '16 at 12:30

Spark Streaming with Scala: is there a way which i can postpone dealing with data to next batch

0 Answers0