scala spark rdd error : java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda

Question

I am a beginner to Scala and Spark.

scala version : 2.12.10

spark version : 3.0.1

I'm trying a very simple spark rdd function in scala.

But I get an error.

(1) build.sbt


scalaVersion := "2.12.10"


name := "hello-world"
organization := "ch.epfl.scala"
version := "1.0"


libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
libraryDependencies +=        "org.apache.spark" %% "spark-sql" % "3.0.1" 
libraryDependencies +=        "org.apache.spark" %% "spark-core" % "3.0.1"

(2) Main.scala

import org.apache.spark.sql.SparkSession
object Main extends App {
  println("Hello, World!")

  implicit val spark = SparkSession.builder()
        .master("spark://centos-master:7077")
        // .master("local[*]")
        .appName("spark-api")
        .getOrCreate()


  val inputrdd = spark.sparkContext.parallelize(Seq(("arth",10), ("arth", 20), ("samuel", 60), ("jack", 65)))
      println("inputrdd : ", inputrdd)
  val mapped = inputrdd.mapValues(x => (x, 1))
  println("mapped : ", mapped)
  mapped.collect.foreach(println)
}

(3) When the error occurred

It seems that an error occurs in the mapped.collect.foreach(println) part.

(4) Error content

21/04/17 20:54:19 INFO DAGScheduler: Job 0 failed: collect at Main.scala:16, took 6.083947 s
[error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, 192.168.0.220, executor 0):

java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda 
to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in 
instance of org.apache.spark.rdd.MapPartitionsRDD

[error]         at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] 
[error] Driver stacktrace:
21/04/17 20:54:19 INFO TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6) on 192.168.0.220, executor 0: java.lang.ClassCastException (cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 7]
21/04/17 20:54:19 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
[error] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, 192.168.0.220, executor 0): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
[error]         at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] 
[error] Driver stacktrace:
[error]         at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
[error]         at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[error]         at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[error]         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[error]         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
[error]         at scala.Option.foreach(Option.scala:407)
[error]         at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
[error]         at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
[error]         at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2139)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2164)
[error]         at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
[error]         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[error]         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[error]         at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
[error]         at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
[error]         at Main$.delayedEndpoint$Main$1(Main.scala:16)
[error]         at Main$delayedInit$body.apply(Main.scala:2)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]         at scala.collection.immutable.List.foreach(List.scala:392)
[error]         at scala.App.main(App.scala:80)
[error]         at scala.App.main$(App.scala:78)
[error]         at Main$.main(Main.scala:2)
[error]         at Main.main(Main.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
[error]         at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)

If I need more libraries or if the code is wrong (but it works fine in spark-shell)....

How to solve it?

You could see what is the type that you get when you do mapped.collect and then try to apply foreach.. — Nikunj Kakadiya, Apr 23 '21 at 04:50
may be u r hitting this bug? https://issues.apache.org/jira/browse/SPARK-29497 — Jay99, Feb 25 '22 at 03:10

score 0 · Answer 1 · answered Mar 28 '22 at 20:04

0

You need to submit your jars to spark so that your code can run on there. spark-shell is hiding all of this from you behind the scenes.

This answer provides better detail https://stackoverflow.com/a/28367602/1810962 with the background.

You can use bin/spark-submit as workaround and provide your local classpath using --class, --jars, and --driver-class-path

answered Mar 28 '22 at 20:04

joseph

2,429
1
22
43

Could you please explain what do you mean by this? >You can use bin/spark-submit as workaround and provide your local classpath using --class, --jars, and --driver-class-path I'm facing the same issue running a java app in kubernetes using `SparkApplication` kind. I've tried putting the jar in the spark classpath, `/opt/spark/jars` and adding the jar in the `SaprkSession`. ``` SparkSession sparkSession = SparkSession .builder() .config("spark.jars", APP_JAR_PATH) .getOrCreate(); ``` – Pablo Cavalieri Jun 26 '23 at 16:02

scala spark rdd error : java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda

1 Answers1