asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Luo <cl...@uci.edu>
Subject Re: PrimaryIndexOperationTracker
Date Tue, 24 Oct 2017 03:18:49 GMT
I was confusing about this problem last quarter, and I remembered (maybe
Ian) told me the same reason that it's mainly used for memory budgeting
(primary and secondary indexes of dataset in the same node). Moreover, it's
not very common to have too many I/O devices on a node, unless it has
multiple disks.

Best regards,
Chen Luo

On Mon, Oct 23, 2017 at 8:12 PM, Sattam Alsubaiee <salsubaiee@gmail.com>
wrote:

> There is a fundamental reason behind that (unless that has changed). All
> partitions of a dataset (primary and secondary indexes) share the memory
> budget for their in-memory components. So they use the same optracker to
> synchronize their operations.
>
>
> Sattam
>
> On Oct 24, 2017 5:59 AM, "Wail Alkowaileet" <wael.y.k@gmail.com> wrote:
>
> > Thanks Abdullah!
> >
> > On Mon, Oct 23, 2017 at 7:15 PM, abdullah alamoudi <bamousaa@gmail.com>
> > wrote:
> >
> > > Hi Wail,
> > > There is no fundamental reason why it is one. In fact, it has been on
> our
> > > todo for a long time to make it one per partition.
> > >
> > > Cheers,
> > > Abdullah.
> > >
> > > > On Oct 23, 2017, at 7:14 PM, Wail Alkowaileet <wael.y.k@gmail.com>
> > > wrote:
> > > >
> > > > Dear devs,
> > > >
> > > > I have a question regarding the opTracker. Currently, we initialize
> one
> > > > opTracker per dataset in every NC.
> > > >
> > > > My question is why it's per dataset not per partition ? Is there a
> > > > transactional constraints for that ?
> > > >
> > > > From what I can see that the opTracker can create a lot of contention
> > in
> > > > case there're many IO devices. For instance, each insert will call
> > > > *LSMHarness.getAndEnterComponents()* [1], which
> > > *synchronize(opTracker). *That
> > > > means (correct me if I'm wrong), insert is going to serialize the
> > > > *enterComponent()* part among partitions.
> > > >
> > > > [1]
> > > > https://github.com/apache/asterixdb/blob/master/hyracks-
> > > fullstack/hyracks/hyracks-storage-am-lsm-common/src/
> > > main/java/org/apache/hyracks/storage/am/lsm/common/impls/
> > > LSMHarness.java#L86
> > > >
> > > > --
> > > >
> > > > *Regards,*
> > > > Wail Alkowaileet
> > >
> > >
> >
> >
> > --
> >
> > *Regards,*
> > Wail Alkowaileet
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message