Thank you very much for taking the time to respond.
My apologies, but at the moment I don't have an example that
I feel comfortable posting. Frankly, I've been struggling with variants
of this for the last two weeks and probably won't be able to work on
this particular issue for a few days.
However, I am intrigued by your comment. You mention "when I close
the fs object inside map/mapPartition etc." Where else can one close the
object? If I don't close it, the output file is generally truncated.
Again, the code seems to work for a few hundred files, then I get these
weird errors. Is this something subtle related to the shipping of the closure that
I'm not aware of?
Can you give a general idea of how you handled this?
Is it necessary to create a custom OutputFormat class?
I was looking at the OutputFormat code and it looks like it also creates
an "fs" object and starts writing, but perhaps there is some
subtle difference in the context?
From: Akhil Das <firstname.lastname@example.org>
To: Michael Albert <email@example.com>
Cc: "firstname.lastname@example.org" <email@example.com>
Sent: Monday, January 5, 2015 1:21 AM
Subject: Re: a vague question, but perhaps it might ring a bell
What are you trying to do? Can you paste the whole code? I used to see this sort of Exception when i close the fs object inside map/mapPartition etc.