kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denny Lee <denny.g....@gmail.com>
Subject Re: Error when processing messages in Windows
Date Wed, 10 Jul 2013 14:59:44 GMT
Thanks very much for digging in!  I was a tad concerned about that
approach but in the process of testing that idea out along with some other
more dramatic ideas ;).  Will keep you updated - thanks again!


On 7/10/13 7:55 AM, "Jun Rao" <junrao@gmail.com> wrote:

>From that link, one workaround is to set the buffer to null and force a
>GC.
>Not sure if that's a good idea though.
>
>Thanks,
>
>Jun
>
>
>On Tue, Jul 9, 2013 at 10:13 PM, Sriram Subramanian <
>srsubramanian@linkedin.com> wrote:
>
>> As far as I am aware it is not possible to resize mapped buffer without
>> unmapping in Windows. W.r.t Java the bug here gives more context on why
>>it
>> does not support synchronous unmap function.
>>
>> http://bugs.sun.com/view_bug.do?bug_id=4724038
>>
>>
>>
>> On 7/9/13 9:54 PM, "Jay Kreps" <jay.kreps@gmail.com> wrote:
>>
>> >The problem appears to be that we are resizing a memory mapped file
>>which
>> >it looks like windows does not allow (which is kind of sucky).
>> >
>> >The offending method is OffsetIndex.resize().
>> >
>> >The most obvious fix would be to first unmap the file, then resize,
>>then
>> >remap it. We can't do this though because Java actually doesn't support
>> >unmapping files (it does this lazily with garbage collection, which
>>really
>> >sucks). In fact as far as I know there is NO way to guarantee an unmap
>> >occurs at a particular time, so if this is correct and windows doesn't
>> >allow resizing then this combination of suckiness means that there is
>>no
>> >way to resize a file that has ever been mapped short of closing the
>> >process.
>> >
>> >I actually don't have access to a windows machine so it is a little
>>hard
>> >for me to test this. The question is whether there is any work around.
>>I
>> >am
>> >happy to change that method but we do need to be able to resize memory
>> >mapped files.
>> >
>> >
>> >
>> >
>> >
>> >
>> >On Tue, Jul 9, 2013 at 9:04 PM, Jun Rao <junrao@gmail.com> wrote:
>> >
>> >> Hmm, not sure what the issue is. Any windows user wants to chime in?
>> >>
>> >> Thanks,
>> >>
>> >> Jun
>> >>
>> >>
>> >> On Tue, Jul 9, 2013 at 9:00 AM, Denny Lee <denny.g.lee@gmail.com>
>> wrote:
>> >>
>> >> > Hey Jun,
>> >> >
>> >> > We've been running into this issue when running perf.Performance as
>> >>per
>> >> > 
>>http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/
>> .
>> >> > When running it using 100K messages, it works fine on Windows with
>> >>about
>> >> > 20-30K msg/s.  But when running it with 1M messages, then the
>>broker
>> >> fails
>> >> > as per the message below.  It does not appear that modifying the
>>JVM
>> >> > memory configurations nor running on SSDs has any effect.   As for
>> >>JVMs -
>> >> > no plug ins and we've tried both 1.6 and OpenJDK 1.7.
>> >> >
>> >> > This looks like a JVM memory map issue on Windows issue - perhaps
>> >>running
>> >> > some System.gc() to prevent the roll?
>> >> >
>> >> > Any thoughts?
>> >> >
>> >> > Thanks!
>> >> > Denny
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On 7/9/13 7:55 AM, "Jun Rao" <junrao@gmail.com> wrote:
>> >> >
>> >> > >A couple of users seem to be able to get 0.8 working on Windows.
>>Any
>> >> thing
>> >> > >special about your Windows environment? Are you using any jvm
>> >>plugins?
>> >> > >
>> >> > >Thanks,
>> >> > >
>> >> > >Jun
>> >> > >
>> >> > >
>> >> > >On Tue, Jul 9, 2013 at 12:59 AM, Timothy Chen <tnachen@gmail.com>
>> >> wrote:
>> >> > >
>> >> > >> Hi all,
>> >> > >>
>> >> > >> I've tried pushing a large amount of messages into Kafka on
>> >>Windows,
>> >> and
>> >> > >> got the following error:
>> >> > >>
>> >> > >> Caused by: java.io.IOException: The requested operation cannot
>>be
>> >> > >>performed
>> >> > >> on a
>> >> > >>  file with a user-mapped section open
>> >> > >>         at java.io.RandomAccessFile.setLength(Native Method)
>> >> > >>         at
>> >>kafka.log.OffsetIndex.liftedTree2$1(OffsetIndex.scala:263)
>> >> > >>         at kafka.log.OffsetIndex.resize(OffsetIndex.scala:262)
>> >> > >>         at
>> >> kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:247)
>> >> > >>         at kafka.log.Log.rollToOffset(Log.scala:518)
>> >> > >>         at kafka.log.Log.roll(Log.scala:502)
>> >> > >>         at kafka.log.Log.maybeRoll(Log.scala:484)
>> >> > >>         at kafka.log.Log.append(Log.scala:297)
>> >> > >>         ... 19 more
>> >> > >>
>> >> > >> I suspect that Windows is not releasing memory mapped file
>> >>references
>> >> > >>soon
>> >> > >> enough.
>> >> > >>
>> >> > >> I wonder if there is any good workaround or solutions for
this?
>> >> > >>
>> >> > >> Thanks!
>> >> > >>
>> >> > >> Tim
>> >> > >>
>> >> >
>> >> >
>> >> >
>> >>
>>
>>



Mime
View raw message