asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taewoo Kim <wangs...@gmail.com>
Subject Is it OK for Hyracks to access the config option?
Date Tue, 24 Oct 2017 16:21:12 GMT
Hello All,

Here is the background. I'm currently fixing an inverted-index search OOM
issue. This could happen because the core inverted-index search module does
not have any budget limit. One might ask that B+-tree and R-tree search
doesn't have any budget limit either. However, a difference is that an
inverted-index has an inverted list (list of primary keys per secondary
key). That is, for a single PK, there can be multiple secondary keys. So,
dealing with this requires additional frames. So, let's assume that we need
to have multiple frames to answer a text query that utilizes an inverted
index.

Here is my issue. We will add one more parameter to the config (e.g.,
compiler.textsearchmemory = 32MB) to control this behavior. The
index-search core module needs to receive and conform to this budget
information. For inverted-index search query case, it's relatively easy. I
can use HyracksTaskContext to pass that since there's a pipeline from
compiler to the search module (IndexSearchOperatorNodePushable). However, I
found an issue regarding the merge case. Merge of an LSM index happens in
Hyracks codebase and index merge code doesn't know about
HyracksTaskContext. So, I talked to Abdullah and he shared a hint about
ServiceContext. In the ServiceContext, there is AppConfig interface and
this contains all config information. And when creating an LSM index
instance, this ServiceContext is accessible.

So, my final question is: is it OK to access AppConfig from Hyracks code by
specifying the exact parameter (e.g., compiler.textsearchmemory) to get
some information? Technically, it's doable since the code exists in
Hyracks. However, most similar parameters are processed in Asterix
codebase, not in  Hyracks codebase. If not, what could be a way to ensure
this?

Thanks for reading!

Best,
Taewoo

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message