Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Improve handling of "Log entry with size X exceeds maximum size" #53

Copy link
Copy link
Closed
@przemekpastuszka

Description

@przemekpastuszka
Issue body actions

Hi,

Lib version: google-cloud-logging==1.15.0

I'm experiencing lots of errors like:

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/google/cloud/logging/handlers/transports/background_thread.py", line 123, in _safely_commit_batch
    batch.commit()
  File "/env/lib/python3.7/site-packages/google/cloud/logging/logger.py", line 383, in commit
    client.logging_api.write_entries(entries, **kwargs)
  File "/env/lib/python3.7/site-packages/google/cloud/logging/_gapic.py", line 126, in write_entries
    partial_success=partial_success,
  File "/env/lib/python3.7/site-packages/google/cloud/logging_v2/gapic/logging_service_v2_client.py", line 477, in write_log_entries
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/env/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/env/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/env/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/env/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Log entry with size 256.6K exceeds maximum size of 256.0K

This traceback doesn't help me identify place where the offending line was logged (I'm using python logging), so I cannot fix it.

I can see at least 3 resolutions to this:

  • allow user to specify that log message can be cropped to max size before sending to Cloud Logging. This way, if someone is ok with it, everything will get (at least partially) logged
  • add to the traceback information where the offending log entry was logged (file name + line number). This way I may be able to identify and fix too long log lines - right now there's no such information.
  • verify the log message size eagerly when logging, i.e. throw exception from methods like logging.info when the message is too long and we won't be able to upload it (so rather than doing the verification in a background thread, do it immediately)

Metadata

Metadata

Assignees

Labels

api: loggingIssues related to the googleapis/python-logging API.Issues related to the googleapis/python-logging API.lang: pythonIssues specific to Python.Issues specific to Python.logging-resiliencepriority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.