Your streaming job may have been seemingly running ok, but the DStream checkpointing must have been failing in the background. It would have been visible in the log4j logs. In 1.4.0, we enabled fast-failure for that so that checkpointing failures dont get hidden in the background.
The fact that the serialization stack is not being shown correctly, is a known bug in Spark 1.4.0, but is fixed in 1.4.1 about to come out in the next couple of days. That should help you to narrow down the culprit preventing serialization.