Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings
Discussion options

neel04
Jul 23, 2022

Hi, I'm using TFDS to stream a dataset from TFrecords stored on a s3 path s3://..... I can confirm they exist, and are non-empty (few terabytes of data in total). This is my log:

2022-07-23 21:47:21.762290: I tensorflow/core/platform/s3/aws_logging.cc:84] Attempting to acquire curl connection.
2022-07-23 21:47:21.762296: I tensorflow/core/platform/s3/aws_logging.cc:71] Connection has been released. Continuing.
2022-07-23 21:47:21.762302: I tensorflow/core/platform/s3/aws_logging.cc:84] Returning connection handle 0x7f616800a000
2022-07-23 21:47:21.762308: I tensorflow/core/platform/s3/aws_logging.cc:84] Obtained connection handle 0x7f616800a000
2022-07-23 21:47:21.773612: I tensorflow/core/platform/s3/aws_logging.cc:84] HTTP/1.1 404 Not Found

2022-07-23 21:47:21.773625: I tensorflow/core/platform/s3/aws_logging.cc:84] x-amz-request-id: 65SDTM7WR91XR7HW

2022-07-23 21:47:21.773637: I tensorflow/core/platform/s3/aws_logging.cc:84] x-amz-id-2: r4e2bY2f07y0QWgZsggOUh48tBowli6SjLZI2PNCaezgkVmbkIoU1VkoevrlLjZEHdCHWCamiwo=

2022-07-23 21:47:21.773645: I tensorflow/core/platform/s3/aws_logging.cc:84] Content-Type: application/xml

2022-07-23 21:47:21.773653: I tensorflow/core/platform/s3/aws_logging.cc:84] Date: Sat, 23 Jul 2022 21:47:21 GMT

2022-07-23 21:47:21.773663: I tensorflow/core/platform/s3/aws_logging.cc:84] Server: AmazonS3

2022-07-23 21:47:21.773671: I tensorflow/core/platform/s3/aws_logging.cc:84]

2022-07-23 21:47:21.773684: I tensorflow/core/platform/s3/aws_logging.cc:84] Returned http response code 404
2022-07-23 21:47:21.773691: I tensorflow/core/platform/s3/aws_logging.cc:84] Returned content type application/xml
2022-07-23 21:47:21.773697: I tensorflow/core/platform/s3/aws_logging.cc:84] Releasing curl handle 0x7f616800a000
2022-07-23 21:47:21.773706: I tensorflow/core/platform/s3/aws_logging.cc:84] Releasing curl handle 0x7f616800a000
2022-07-23 21:47:21.773712: I tensorflow/core/platform/s3/aws_logging.cc:84] Notified waiting threads.
2022-07-23 21:47:21.773725: I tensorflow/core/platform/s3/aws_logging.cc:84] Request returned error. Attempting to generate appropriate error codes from response
2022-07-23 21:47:21.773735: E tensorflow/core/platform/s3/aws_logging.cc:77] HTTP response code: 404
Exception name:
Error message: No response body.
5 response headers:
content-type : application/xml
date : Sat, 23 Jul 2022 21:47:21 GMT
server : AmazonS3
x-amz-id-2 : r4e2bY2f07y0QWgZsggOUh48tBowli6SjLZI2PNCaezgkVmbkIoU1VkoevrlLjZEHdCHWCamiwo=
x-amz-request-id : 65SDTM7WR91XR7HW
2022-07-23 21:47:21.773750: W tensorflow/core/platform/s3/aws_logging.cc:74] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2022-07-23 21:47:21.773760: I tensorflow/core/platform/s3/aws_logging.cc:84] Server time is Sat, 23 Jul 2022 21:47:21 GMT, while client time is Sat, 23 Jul 2022 21:47:21 GMT
2022-07-23 21:47:21.773775: E tensorflow/core/platform/s3/aws_logging.cc:77] Transfer handle [F66BACBE-4F50-4351-B448-BF8064BA7DA7] Failed to get download parts information for object in Bucket: [s-laion] with Key: [ssd-videos/new_tfrecs/bdd100k/1.0.0/bdd100k-train.tfrecord-00056-of-47044] HTTP response code: 404
Exception name:
Error message: No response body.
5 response headers:
content-type : application/xml
date : Sat, 23 Jul 2022 21:47:21 GMT
server : AmazonS3
x-amz-id-2 : r4e2bY2f07y0QWgZsggOUh48tBowli6SjLZI2PNCaezgkVmbkIoU1VkoevrlLjZEHdCHWCamiwo=
x-amz-request-id : 65SDTM7WR91XR7HW
2022-07-23 21:47:21.773786: I tensorflow/core/platform/s3/aws_logging.cc:71] Transfer handle ID [F66BACBE-4F50-4351-B448-BF8064BA7DA7] Updated handle status from [NOT_STARTED] to [FAILED].

If anyone has any idea how to resolve this, please help! 🙏
You must be logged in to vote

Replies: 1 comment · 4 replies

Comment options

Conchylicultor
Jul 25, 2022
Maintainer

It would be helpful to get a reproductible minimal code snippet which correspond to the above stacktrace which uses only TF.
You must be logged in to vote
4 replies
@Conchylicultor
Comment options

Conchylicultor Jul 25, 2022
Maintainer

I also think this is a https://github.com/tensorflow/tensorflow issue. Not a TFDS issue.
@neel04
Comment options

neel04 Jul 25, 2022
Author

I haven't fully triangulated the bug yet, but its definitely with TFDS. The problem is that somehow the internal modules which glob the filepaths on calling tfds.load() are NOT globbing the latest filepaths - I'm not familiar with the internals but I assume it might be caching it somewhere due to the dataset name provided as a parameter staying the same.

I'll do a deep dive in the code, but if you happen to have any idea about it (or a code snippet you think might be troublesome) that would be nice!
@neel04
Comment options

neel04 Jul 25, 2022
Author

For a reproducible snippet, you'd need an S3 bucket with dummy files, say named:

random-test.tfrecord-00000-of-00001
random-train.tfrecord-00000-of-00001

Perform a:

import tensorflow_datasets as tfds
data = tfds.load(name='random', ... ) #S3 path goes here

Then simply delete those dummy TFRecords for something different:

random-test.tfrecord-00000-of-00002
random-test.tfrecord-00001-of-00002
random-train.tfrecord-00000-of-00002
random-train.tfrecord-00001-of-00002

Because the filepath changes (00002 instead of 00001) the paths TFDS got from my dir were the same, but they'd been updated with more files - so there were more TFRecords as a result x-of-y the value of y changed - and not by much, so its a subtle change..
@neel04
Comment options

neel04 Jul 25, 2022
Author

I think the reason is that once you update the dataset, one has to NOT use the same version 1.0.0 like I did. If this indeed reproduces the issue, then it would be nice if a warning can be put!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
2 participants
@neel04 @Conchylicultor
Morty Proxy This is a proxified and sanitized view of the page, visit original site.