Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Low default chunk size slows GCS downloads #2222

Copy link
Copy link

Description

@davidn
Issue body actions

Steps to reproduce:

  1. Create multi-gb file gs://your_bucket/obj
  2. Test gsutil speed with "gsutil cp gs://your_bucket/obj /local/copy"
  3. download via gcloud.storage.Blob.download_to_filename (see sample script below)

Expected result: similar timing

Actual result on my machine: gsutil downloads 4.6GB in 1min14sec, gcloud.storage takes 20min.

If I add blob.chunk_size=1<<30 prior to download_to_filename, gcloud.storage takes the same time as gsutil to the second.

Sample script:

#!/usr/bin/env python
import httplib2
httplib2.debuglevel=4
import gcloud.storage

bucket_id = 'BUCKET'
blob_id = 'OBJECT'
url = 'gs://%s/%s' % (bucket_id, blob_id)
project = 'dnewgas-test-1'
client = gcloud.storage.client.Client(project)
bucket = client.get_bucket(bucket_id)
blob = bucket.get_blob(blob_id)
//blob.chunk_size=1<<30
blob.download_to_filename('/tmp/down')
Reactions are currently unavailable

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.Issues related to the Cloud Storage API.performancepriority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.type: processA process-related concern. May include testing, release, or the like.A process-related concern. May include testing, release, or the like.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.