Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

BigQuery: obscure exception when writing 0 bytes #2249

Copy link
Copy link

Description

@NicolaSpreafico
Issue body actions

I wrote this code in an App Engine standard (Java7) application which writes a set of rows in a BigQuery table

My code started from here: https://cloud.google.com/bigquery/loading-data-post-request#bigquery-import-file-java

private static final Charset UTF8 = Charset.forName("UTF-8");

// Instantiates a client
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

// Create the reference for the Table
TableId tableId = TableId.of(project, dataset, table);

WriteChannelConfiguration writeChannelConfiguration = WriteChannelConfiguration
    .newBuilder(tableId)
    .setFormatOptions(FormatOptions.json())
    .build();

// Open a new channel for write
TableDataWriteChannel writer = bigquery.writer(writeChannelConfiguration);

// Retrieves the object from DB
List<T> objects = ....;

// Write data to writer
try (OutputStream stream = Channels.newOutputStream(writer)) {
    // Access to iterator of the objects
    Iterator<T> iterator = objects.iterator();
    
    while (iterator.hasNext()) {
        // Retrieve the next element
        T object = iterator.next();
    
        // Serialize the object in json format
        JsonObject json = serialize(object);
    
        // Write the json to the stream
        stream.write(json.toString().getBytes(UTF8));
    
        // Increment the counter
        index++;
    
        // Check if additional records are available
        if (iterator.hasNext()) {
            stream.write("\n".getBytes(UTF8));
        }
    }
}

// Get job status
Job job = writer.getJob();

// Wait for job to end
try {
    job = job.waitFor();
} catch (InterruptedException | TimeoutException e) {
    throw new IOException("WaitFor error", e);
}

// Retrieve job statistics
LoadStatistics stats = job.getStatistics();
LOG.log(Level.INFO, "Executed Job: {0}", new Object[] {stats});

In localhost works fine, w/o problems. All the rows are written and no errors raised.

But after I ran this code online I got this error

{"message":"400\nFailed to parse Content-Range header.","errors":[{"message":"400\nFailed to parse Content-Range header."},{"message":"400\nFailed to parse Content-Range header."}]}

Failed to parse Content-Range header.
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:86)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:466)
	at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:57)
	at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:54)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
	at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:53)
	at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:160)
	at java.nio.channels.Channels$1.close(Channels.java:178)
Caused by: com.google.api.client.http.HttpResponseException: 400
Failed to parse Content-Range header.
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
	at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:449)
	... 69 more

Can this library be used on App Engine standard environment? I'm asking this because for the PubSub counterpart there is a warning

However, we recommend using the older Google APIs Client Libraries if running on Google App Engine standard environment

that I cannot see on BigQuery page.

My goal here is loading a bunch of rows directly inside the table w/o using a temporary file, loading the data directly from the application memory.

I enabled the full Log of HTTP Transport, here is a part of it

com.google.api.client.http.HttpRequest execute: -------------- REQUEST  -------------- (HttpRequest.java:962)
PUT https://www.googleapis.com/upload/bigquery/v2/projects/noovle-erp-orchestrator-dev/jobs?uploadType=resumable&upload_id=AEnB2Ur7N3UM_EkF6fpM9NQFmKhsEtS1y2IsEXM0DFjnPIAn6udvTx6gA5dWDzbfwscWkbLUXy9llTBhssjG8n-llBgSlYs-2g
Accept-Encoding: gzip
Authorization: <Not Logged>
Content-Range: bytes 0--1/0
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
x-goog-api-client: gl-java/1.7.0-google-v6 gccl/0.20.1-beta
Content-Length: 0

Content-Length is 0 and Content-Range is 0 as well. Maybe this type of in-memory load does not work because I do not have in advance the size of the full data?
If so, why in localhost works fine?

Reactions are currently unavailable

Metadata

Metadata

Labels

api: bigqueryIssues related to the BigQuery API.Issues related to the BigQuery API.priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.