This directory provides implementations of datasets used to evaluate the MLPerf v1.0 benchmark.These implementations are initially developed to be used with TFLite. Other backends may need to extend them.

If you need help obtaining the ImageNet dataset, contact the chairs for suggestions. For testing purpose, you may download a substitute dataset from https://github.com/mlcommons/mobile_models/tree/main/v0_7/datasets

Download the COCO 2017 dataset from http://cocodataset.org/#download and the script upscale_coco.py. Then use the script to process the images:

python upscale_coco.py --inputs /path-to-coco/ --outputs /output-path/ --size 300 300

The ground truth file is coco_val_full.pbtxt. If you want to use a subset of images, remember to use the first N images which appear in the file instances_val2017.json. Note that the order of images in this file and the order of images under the images directory are not the same.

Download the Squad v1.1 evaluation set and the vocab file. Then, you can generate the groundtruth and input tfrecord files for running inference as below:

1. Install dependencies:

   pip install tensorflow tensorflow_hub

2. Download other dependencies from google-research and add its path to PYTHONPATH:

   TMP_DIR=<your tmp directory>
   pushd $TMP_DIR
   curl -o google-research.tar.gz -L https://github.com/google-research/google-research/archive/256f678d1aeb7a4527031c8dd2f4a2c9f3833f93.tar.gz
   tar -xzf google-research.tar.gz --transform s/google-research-256f678d1aeb7a4527031c8dd2f4a2c9f3833f93/google-research/
   export PYTHONPATH="${PYTHONPATH}:`pwd`/google-research"
   popd

3. Generate tfrecord files for inference:

   python cpp/datasets/squad_utils/generate_tfrecords.py \
     --vocab_file=<path to vocab.txt> \
     --predict_file=<path to dev-v1.1.json> \
     --output_dir=<output dir> \
     --max_seq_length=384 \
     --doc_stride=128 \
     --max_query_length=64

There are default tfrecord files at https://github.com/mlcommons/mobile_models/tree/main/v1_0/datasets generated with above default parameters. By default, the app will use a mini version of the dataset with 160 random questions. To evaluate using the full dataset, you need to replace squad_eval_mini.tfrecord by squad_eval.tfrecord in the tasks_v2.pbtxt file or in the /storage/emulated/0/mlperf_datasets/tasks_v2.pbtxt file if you use 'user-defined' tasks file.

1. follow DeepLab's instruction to download the ADE20K dataset

2. prepare 512x512 images and and ground truth file with something like the following

   using .jpg inputs:

   import os
   import tensorflow as tf
   import deeplab.input_preprocess
   from PIL import Image as Image
   
   tf.enable_eager_execution()
   
   home = os.getenv("HOME")
   ADE20K_PATH = home + '/tf-models/research/deeplab/datasets/ADE20K/ADEChallengeData2016/'
   
   for i in range(1, 2001):
       image_jpeg = ADE20K_PATH+f'images/validation/ADE_val_0000{i:04}.jpg'
       label_png = ADE20K_PATH+f'annotations/validation/ADE_val_0000{i:04}.png'
   
       image_jpeg_data = tf.io.read_file(image_jpeg)
       image_tensor = tf.io.decode_jpeg(image_jpeg_data)
       label_png_data = tf.io.read_file(label_png)
       label_tensor = tf.io.decode_png(label_png_data)
       o_image, p_image, p_label = deeplab.input_preprocess.preprocess_image_and_label(image_tensor, label_tensor, 512, 512, 512, 512, is_training=False)
   
       target_image_jpeg = f'/tmp/ade20k_512/images/validation/ADE_val_0000{i:04}.jpg'
       target_label_png = f'/tmp/ade20k_512/annotations/ADE_val_0000{i:04}.png'
   
       resized_image = Image.fromarray(tf.reshape(tf.cast(p_image, tf.uint8), [512, 512, 3]).numpy())
       resized_image.save(target_image_jpeg, quality=100, subsampling=0))
   
       resized_label = Image.fromarray(tf.reshape(tf.cast(p_label, tf.uint8), [512, 512]).numpy())
       resized_label.save(target_label_png)

   using .png inputs:

   home = os.getenv("HOME")
   ADE20K_PATH = home + '/tf-models/research/deeplab/datasets/ADE20K/ADEChallengeData2016/'
   
   for i in range(1, 2001):
       image_jpeg = ADE20K_PATH+f'images/validation/ADE_val_0000{i:04}.jpg'
       label_png = ADE20K_PATH+f'annotations/validation/ADE_val_0000{i:04}.png'
       # print(image_jpeg)
       image_jpeg_data = tf.io.read_file(image_jpeg)
       image_tensor = tf.io.decode_jpeg(image_jpeg_data)
       label_png_data = tf.io.read_file(label_png)
       label_tensor = tf.io.decode_png(label_png_data)
       o_image, p_image, p_label = deeplab.input_preprocess.preprocess_image_and_label(image_tensor, label_tensor, 512, 512, 512, 512, is_training=False)
   
       target_image_png = f'/tmp/ade20k_512/images/validation/ADE_val_0000{i:04}.png'
       target_label_png = f'/tmp/ade20k_512/annotations/ADE_val_0000{i:04}.png'
   
       resized_image = Image.fromarray(tf.reshape(tf.cast(p_image, tf.uint8), [512, 512, 3]).numpy())
       resized_image.save(target_image_png)
   
       resized_label = Image.fromarray(tf.reshape(tf.cast(p_label, tf.uint8), [512, 512]).numpy())
       resized_label.save(target_label_png)

3. Build command line tool to test performance and accuracy on x86 host

   build  --cxxopt='--std=c++14' --host_cxxopt='--std=c++14' --copt=-march=native //cpp/binary:main

4. test with the command line tool

   ./bazel-bin/cpp/binary/main tflite ade20k \
     --mode=PerformanceOnly \
     --output_dir=/tmp/test_output \
     --model_file=/tmp/freeze_quant_ops16_32c_clean.tflite \
     --images_directory=/tmp/ade20k_512/images/validation  \
     --ground_truth_directory=/tmp/ade20k_512/annotations/raw  \
     --num_threads=4

preparing datasets and models

1. dataset:

   1. rotate 1080x1920 ones to 1920x1080 so that all the images are 1920x1080

   2. convert all 1920x1080 .png images to raw rgb files with .rgb8 suffix, because so far .png is not supported by the preprocessing code and .jpg supported by the preprocessing code doesn't support lossless compression

   3. also generate scaled-down 960x540 images

2. model:

   1. generate a model with 960x540 input and 1920x1080 output

3. test with command line on Android device:

   Build the main command line program and tflite backend.

   bazel build --config android_arm64 -c opt \
     flutter/cpp/binary:main mobile_back_tflite:tflitebackend

   Push them to the target Android device

   adb push bazel-bin/flutter/cpp/binary/main /data/local/tmp/sr/main_sr
   adb push bazel-bin/mobile_back_tflite/cpp/backend_tflite/libtflitebackend.so \
     /data/local/tmp/sr/

   Assuming we have the dataset and the model on the devices at /data/local/tmp/sr/dataset/ and /data/local/tmp/edsr/tflite/pl_f32b5.tflite

   adb shell /data/local/tmp/sr/main_sr external snusr --mode=PerformanceOnly \
     --output_dir=/data/local/tmp/sr_output \
     --model_file=/data/local/tmp/edsr/tflite/pl_f32b5.tflite \
     --images_directory=/data/local/tmp/sr/dataset/LR_raw \
     --ground_truth_directory=/data/local/tmp/sr/dataset/HR_raw \
     --lib_path=/data/local/tmp/sr/libtflitebackend.so

Or we can test this on a host machine too.

On a x86 machine running Ubuntu 22.04,

bazel build -c opt flutter/cpp/binary:main mobile_back_tflite:tflitebackend \
      --host_cxxopt=-std=c++14 --cxxopt=-std=c++14 --copt=-march=native

Assuming we have the model and images in right places,

bazel-bin/flutter/cpp/binary/main external snusr --mode=AccuracyOnly \
  --output_dir=/tmp/sr_output \
  --model_file=/tmp/tflite/pl_f32b7.tflite \
  --images_directory=/tmp/sr/dataset/LR_raw \
  --ground_truth_directory=/tmp/sr/dataset/HR_raw \
  --lib_path=bazel-bin/mobile_back_tflite/cpp/backend_tflite/libtflitebackend.so