asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xikui Wang <xik...@uci.edu>
Subject Data flushing in AbstractOneInputOneOutputOneFramePushRuntime
Date Fri, 08 Mar 2019 01:26:01 GMT
Hi Devs,

I'm working on an experiment, in which I have a socket feed receives data
at a very low rate (1 record/second). In data feeds, when data is not
coming fast enough, the flow controller will try to force a flush before it
heads into waiting. However, I found the data was not flushed as I expected
in my experiment.

After some investigation, I pinpointed a "potential suspect". In the
"flushAndReset" method of the AbstractOneInputOneOutputOneFramePushRuntime
class [1], the record appender writes data records into its writer, but
that writer is not flushed. Thus, the data records are not flushed as
expected, even when the "flushFramesRapidly" flag is on. After adding
a writer.flush() call, everything runs as expected.

This feels like a bug to me. Since this class touches many runtime
factories, and it may have impacts on the performance, I want to double
check with you guys before I submit the fix. Thoughts?

[1]
org/apache/hyracks/algebricks/runtime/operators/base/AbstractOneInputOneOutputOneFramePushRuntime.java:74

Best,
Xikui

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message