flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Perez (Jira)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-22436) twitter datastream connector hangs, CRLF expected at end of chunk
Date Sat, 01 May 2021 13:49:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-22436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Perez updated FLINK-22436:
--------------------------------
    Description: 
Sorry for selecting Connectors/Common, Twitter didn't show up in the Connectors / "X" list,
I'm not sure why that is.

 

in addition to item FLINK-22435 I found this exception as well when the recent version (1.12)
twitter connector just hangs. 

 

basically I am following the example here:

[https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/twitter/TwitterExample.java]

 

which just seems to run indefinitely but not have any data throughput/results or exceptions.

 

task manager stdout (not logs) looks something like this:

 
{code:java}
WARNING: Please consider reporting this to the maintainers of org.apache.flink.shaded.akka.org.jboss.netty.util.internal.ByteBufferUtilWARNING:
Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING:
All illegal access operations will be denied in a future releaseMay 01, 2021 8:37:04 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables
closeWARNING: IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
May 01, 2021 8:39:06 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
May 01, 2021 8:49:48 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
{code}
 

So the part that stood out to me was "CRLF expected at end of chunk", then stacktrace.

I'm not sure which is the cause of twitter connector not working between this and FLINK-22435
but I'm reporting both just in case.

 

I can provide more info if needed but my setup is really basic/simple.

 

  was:
Sorry for selecting Connectors/Common, Twitter didn't show up in the Connectors / "X" list,
I'm not sure why that is.

 

in addition to item FLINK-22435 I found this exception as well when the recent version (1.12)
twitter connector just hangs. 

 

basically I am following the example here:

[https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/twitter/TwitterExample.java]

 

which just seems to run indefinitely but not have any data throughput/results or exceptions.

 

task manager stdout (not logs) looks something like this:

 
{code:java}
WARNING: Please consider reporting this to the maintainers of org.apache.flink.shaded.akka.org.jboss.netty.util.internal.ByteBufferUtilWARNING:
Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING:
All illegal access operations will be denied in a future releaseMay 01, 2021 8:37:04 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables
closeWARNING: IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
May 01, 2021 8:39:06 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
May 01, 2021 8:49:48 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING:
IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
{code}
and this just seems to repeat in the log.

I can provide more info if needed but my setup is really basic/simple.

 


> twitter datastream connector hangs, CRLF expected at end of chunk
> -----------------------------------------------------------------
>
>                 Key: FLINK-22436
>                 URL: https://issues.apache.org/jira/browse/FLINK-22436
>             Project: Flink
>          Issue Type: Bug
>          Components: API / DataStream, Connectors / Common
>         Environment: My environment is just simple local flink env, using the twitter
connector example.
>            Reporter: Jason Perez
>            Priority: Major
>
> Sorry for selecting Connectors/Common, Twitter didn't show up in the Connectors / "X"
list, I'm not sure why that is.
>  
> in addition to item FLINK-22435 I found this exception as well when the recent version
(1.12) twitter connector just hangs. 
>  
> basically I am following the example here:
> [https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/twitter/TwitterExample.java]
>  
> which just seems to run indefinitely but not have any data throughput/results or exceptions.
>  
> task manager stdout (not logs) looks something like this:
>  
> {code:java}
> WARNING: Please consider reporting this to the maintainers of org.apache.flink.shaded.akka.org.jboss.netty.util.internal.ByteBufferUtilWARNING:
Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING:
All illegal access operations will be denied in a future releaseMay 01, 2021 8:37:04 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables
closeWARNING: IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
> May 01, 2021 8:39:06 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables
closeWARNING: IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
> May 01, 2021 8:49:48 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables
closeWARNING: IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException:
CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at
org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at
org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at
org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at
org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at
java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at
org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at
com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at
java.base/java.lang.Thread.run(Thread.java:832)
> {code}
>  
> So the part that stood out to me was "CRLF expected at end of chunk", then stacktrace.
> I'm not sure which is the cause of twitter connector not working between this and FLINK-22435
but I'm reporting both just in case.
>  
> I can provide more info if needed but my setup is really basic/simple.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message