spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Spark with GCS Connector - Rate limit error
Date Tue, 11 Aug 2015 08:44:53 GMT
There's a daily quota and a minutely quota, you could be hitting those. You
can ask google to increase the quota for the particular service. Now, to
reduce the limit from the spark side, you can actually to a re-partition to
a smaller number before doing the save. Another way to use the local file
system rather than the GFS would be to set the spark.local.dir and
spark.worker.dir configurations for the job.

Thanks
Best Regards

On Mon, Aug 10, 2015 at 4:39 PM, Oren Shpigel <oren@yowza3d.com> wrote:

> Hi,
> I'm using Spark on a Google Compute Engine cluster with the Google Cloud
> Storage connector (instead of HDFS, as recommended  here
> <https://cloud.google.com/hadoop/google-cloud-storage-connector#benefits>
> ), and get a lot of "rate limit" errors, as added below.
> The errors relate to temp files (in a folder called _temporary), and not to
> my input/output of the program.
>
> Is there a way to control the read/write rate of Spark?
> Is there a way to increase the rate limit for my Google Project?
> Is there a way to use local Hard-Disk for temp files that don't have to be
> shared with other slaves?
> Or anyone knows or thinks of any other solution for that?
>
> Thanks,
> Oren
>
> ------------------------
>
> java.io.IOException: Error inserting: bucket: *****, object: *****
>   at
>
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.wrapException(GoogleCloudStorageImpl.java:1600)
>   at
>
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl$3.run(GoogleCloudStorageImpl.java:475)
>   at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by:
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 429 Too
> Many Requests
> {
>   "code" : 429,
>   "errors" : [ {
>     "domain" : "usageLimits",
>     "message" : "The total number of changes to the object ***** exceeds
> the
> rate limit. Please reduce the rate of create, update, and delete
> requests.",
>     "reason" : "rateLimitExceeded"
>   } ],
>   "message" : "The total number of changes to the object ***** exceeds the
> rate limit. Please reduce the rate of create, update, and delete requests."
> }
>   at
>
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
>   at
>
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
>   at
>
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
>   at
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
>   at
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>   at
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
>   at
>
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl$3.run(GoogleCloudStorageImpl.java:472)
>   ... 3 more
>
> ------------------------
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-with-GCS-Connector-Rate-limit-error-tp24194.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message