phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumit Nigam <sumit_o...@yahoo.com>
Subject Re: simple commands that mutate a large number of rows
Date Tue, 15 Sep 2015 16:09:18 GMT
So, if I set this property to a very small value, then it will not be honored if auto-commit
is set to true? Is it that in case of autocommit, the engine figures out its own buffer size
or we can influence it through some property again?
      From: James Taylor <jamestaylor@apache.org>
 To: "user@phoenix.apache.org" <user@phoenix.apache.org> 
Cc: Sumit Nigam <sumit_only@yahoo.com> 
 Sent: Tuesday, September 15, 2015 9:18 PM
 Subject: Re: simple commands that mutate a large number of rows
   
Yes, when using auto commit, the phoenix.mutate.maxSiiz config property has no effect.

On Tuesday, September 15, 2015, James Heather <james.heather@mendeley.com> wrote:



I can confirm that setting auto-commit to true avoided the problem.That's for this particular
query, though. I don't know whether it would do if the select query hit a different table.
Probably that would mean it would execute client side, and the error would come back.On 15
Sep 2015 16:12, "Sumit Nigam" <sumit_only@yahoo.com> wrote:

Hi James,
Is it right to assume that with auto-commit set to true, the mutate maxSize being exceeded
error would not occur? This should be because now server side does the commit automatically
when the batch size is reached/ buffered.
Thanks,Sumit
      From: James Taylor <jamestaylor@apache.org>
 To: "user@phoenix.apache.org" <user@phoenix.apache.org> 
 Sent: Tuesday, September 15, 2015 7:37 PM
 Subject: Re: simple commands that mutate a large number of rows
   
That config setting (phoenix.mutate.maxSiize) is just a safety valve to prevent out of memory
errors and may be set to whatever you like. However, if you're going to just turn around and
do a commit after running your upsert statement, performance will improve if you turn on auto
commit instead (connection.setAutoCommit(true)), as that'll prevent double buffering and enable
execution completely on the server side when possible.
Thanks,James

On Tuesday, September 15, 2015, James Heather <james.heather@mendeley.com> wrote:



  I found today that I can't execute this:
 
 UPSERT INTO loadtest.testing (id, firstname, lastname) SELECT NEXT VALUE FOR loadtest.testing_id_seq,
firstname, lastname FROM loadtest.testing


 when the table has more than 500,000 rows in it ("MutationState size of 512000 is bigger
than max allowed size of 500000").
 
 Is there a reason for the restriction, or a way of lifting it so that things get processed
in batches?
 
 Does this also mean that if I had a large table, and I wanted to (say) null out one of the
columns, I wouldn't be able to do it in one hit because I'd be changing too many rows?
 
 James
  


   



  
Mime
View raw message