asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Maxon <>
Subject Re: Is it OK for Hyracks to access the config option?
Date Tue, 24 Oct 2017 20:35:53 GMT
The ini-style config should be fine to access from Hyracks-level, it's
modular. Some of the properties are already accessed from within
Hyracks for the most part (e.g. IO devices, etc).

On Tue, Oct 24, 2017 at 1:20 PM, Taewoo Kim <> wrote:
> Never mind about the first question. The problem becomes simpler (but still
> couldn't find the solution) after talking to Chen Luo. Thanks.
> Let's say the inverted index search module Blah1 (Actually SearchResult and
> FixedSizeElementInvertedListCursor in case you are interested in). For this
> Blah1 module to generate intermediate and final result work properly, the
> caller needs to pass three parameters (frameSize, how to
> allocate/deallocate buffers, and the maximum number of frames that Blah1
> can have). Blah1 can be called in two ways.
> 1) Search using HyracksTaskCotext (class name: Task)
> 2) Search initiated by merge (class name: AsynchronousScheduler)
> 1) and 2) are independent processes so it looks like hard to associate 1)
> in 2) or 2) in 1). So, for Blah1, it needs to consider 1) as default and 2)
> as a special case or vice versa. Any suggestions? Again, for B+ tree, R
> tree, this problem doesn't exist since there is no intermediate result
> calculation.
> Best,
> Taewoo
> On Tue, Oct 24, 2017 at 9:21 AM, Taewoo Kim <> wrote:
>> Hello All,
>> Here is the background. I'm currently fixing an inverted-index search OOM
>> issue. This could happen because the core inverted-index search module does
>> not have any budget limit. One might ask that B+-tree and R-tree search
>> doesn't have any budget limit either. However, a difference is that an
>> inverted-index has an inverted list (list of primary keys per secondary
>> key). That is, for a single PK, there can be multiple secondary keys. So,
>> dealing with this requires additional frames. So, let's assume that we need
>> to have multiple frames to answer a text query that utilizes an inverted
>> index.
>> Here is my issue. We will add one more parameter to the config (e.g.,
>> compiler.textsearchmemory = 32MB) to control this behavior. The
>> index-search core module needs to receive and conform to this budget
>> information. For inverted-index search query case, it's relatively easy. I
>> can use HyracksTaskContext to pass that since there's a pipeline from
>> compiler to the search module (IndexSearchOperatorNodePushable). However,
>> I found an issue regarding the merge case. Merge of an LSM index happens in
>> Hyracks codebase and index merge code doesn't know about
>> HyracksTaskContext. So, I talked to Abdullah and he shared a hint about
>> ServiceContext. In the ServiceContext, there is AppConfig interface and
>> this contains all config information. And when creating an LSM index
>> instance, this ServiceContext is accessible.
>> So, my final question is: is it OK to access AppConfig from Hyracks code
>> by specifying the exact parameter (e.g., compiler.textsearchmemory) to get
>> some information? Technically, it's doable since the code exists in
>> Hyracks. However, most similar parameters are processed in Asterix
>> codebase, not in  Hyracks codebase. If not, what could be a way to ensure
>> this?
>> Thanks for reading!
>> Best,
>> Taewoo

View raw message