Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Make gcloud.storage use a streaming HTTP library for downloads #11

Copy link
Copy link
Closed
@jgeewax

Description

@jgeewax
Issue body actions

Right now we're using httplib2 because of oauth2client working really nicely with that, however httplib2 doesn't support a streaming download.

This means, a .request() will load the entire response content into memory, which won't work at all for a big file.

To get around this, we chunk the file up using the Range HTTP header and request pieces of it until the end, however the better way to handle this is with a streaming HTTP library:

stream = <send http request>
with open('output.txt', 'wb') as output:
  data = stream.read(CHUNK_SIZE)
  while data:
    output.write(data)
    data = stream.read(CHUNK_SIZE)

To make this work we need to:

  • use a library that does support streaming
  • update oauth2client to work with that library

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.Issues related to the Cloud Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.