examples/cli/README.md

# Beyond Get Started CLI Example

If you already walked through the [Get Started Guide](/GetStarted.md), this
example begins with the same training steps and then continues with more examples of
using CLI commands to work with that trained model.

The following example walks through a full workflow using the Intel Transfer Learning
Tool CLI to train a model, and then benchmark, quantize, and optimize the
trained model. It uses a TensorFlow image classification model, but the
same commands and concepts can be applied when working with other frameworks
and use cases.

Use `tlt --help` to see the list of CLI commands. More detailed information on
each command can be found using `tlt <command> --help` (such as `tlt train --help`).

## List the Available Models

Use the `tlt list` command to see a list of available models for each framework.
Use the `--use-case` flag to limit the list to models for a particular use case.
```
tlt list models --use-case image_classification
```
```
------------------------------
IMAGE CLASSIFICATION
------------------------------
alexnet (pytorch)
convnext_base (pytorch)
convnext_large (pytorch)
convnext_small (pytorch)
convnext_tiny (pytorch)
densenet121 (pytorch)
densenet161 (pytorch)
densenet169 (pytorch)
densenet201 (pytorch)
efficientnet_b0 (pytorch)
efficientnet_b0 (tensorflow)
efficientnet_b1 (pytorch)
efficientnet_b1 (tensorflow)
efficientnet_b2 (pytorch)
efficientnet_b2 (tensorflow)
efficientnet_b3 (pytorch)
efficientnet_b3 (tensorflow)
efficientnet_b4 (pytorch)
efficientnet_b4 (tensorflow)
...
```

## Train a Model

For this example, we use the TensorFlow flowers dataset. Download and extract
the dataset by following this example:
```
# Create a directory for the dataset to be downloaded
DATASET_DIR=/tmp/dataset
mkdir -p ${DATASET_DIR}

# Download and extract the dataset
wget -P ${DATASET_DIR} https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz
tar -xzf ${DATASET_DIR}/flower_photos.tgz -C ${DATASET_DIR}

# Set the DATASET_DIR to the extracted images folder
DATASET_DIR=${DATASET_DIR}/flower_photos

# Supress debug information from TensorFlow 2.12
TF_CPP_MIN_LOG_LEVEL=2
```

After the dataset directory is ready, use the `tlt train` command to train one of the models from
`tlt list`. In this example, we use the TensorFlow ResNet50v1.5 model. Make sure to specify
your own file path for the `output-dir`. The `dataset-dir` should point to the extracted dataset folder.
```
tlt train -f tensorflow --model-name resnet_v1_50 --dataset-dir ${DATASET_DIR} --output-dir /tmp/output
```
```
Model name: resnet_v1_50
Framework: tensorflow
Training epochs: 1
Dataset dir: /tmp/dataset/flower_photos
Output directory: /tmp/output
Found 3670 files belonging to 5 classes.
...
Model: "sequential"
_________________________________________________________________
Layer (type)                Output Shape              Param #
=================================================================
keras_layer (KerasLayer)    (None, 2048)              23561152
dense (Dense)               (None, 5)                 10245
=================================================================
Total params: 23,571,397
Trainable params: 10,245
Non-trainable params: 23,561,152
_________________________________________________________________
Checkpoint directory: /tmp/output/resnet_v1_50_checkpoints
86/86 [==============================] - 24s 248ms/step - loss: 0.4600 - acc: 0.8438
Saved model directory: /tmp/output/resnet_v1_50/1
```

The `tlt train` command evaluates the model after training completes. The loss and
accuracy values are printed toward the end of the console output, along with the
location where the trained model has been saved (including a new numbered folder for
each run).

A trained model can also be evaluated using the `tlt eval` command:
```
tlt eval --model-dir /tmp/output/resnet_v1_50/1 --dataset-dir ${DATASET_DIR}
```

## Benchmark the Trained Model

Benchmark the performance of the trained model using `tlt benchmark`.  Make sure
to specify your own file paths for `model-dir` (including the numbered folder).
The `dataset-dir` should point to the extracted dataset folder.
```
tlt benchmark --model-dir /tmp/output/resnet_v1_50/1 --dataset-dir ${DATASET_DIR} --batch-size 512
```
```
Model directory: /tmp/output/resnet_v1_50/1
Dataset directory: /tmp/dataset/flower_photos
Batch size: 512
Model name: resnet_v1_50
Framework: tensorflow
...
performance mode benchmark result:
2022-06-28 10:22:10 [INFO] Batch size = 512
2022-06-28 10:22:10 [INFO] Latency: 3.031 ms
2022-06-28 10:22:10 [INFO] Throughput: 329.878 images/sec
   ```

## Quantize the Model

Perform post-training quantization using the [Intel® Neural
Compressor](https://intel.github.io/neural-compressor) using the `tlt quantize`
command. Make sure to specify your own file paths for `model-dir` (including the
numbered folder), `dataset-dir`, and `output-dir`.  The quantized model will be
saved to the output directory.
```
tlt quantize --model-dir /tmp/output/resnet_v1_50/1 --dataset-dir ${DATASET_DIR} --batch-size 512 \
--accuracy-criterion 0.01 --output-dir /tmp/output
```
```
Model directory: /tmp/output/resnet_v1_50/1
Dataset directory: /tmp/dataset/flower_photos
Accuracy criterion: 0.01
Exit policy timeout: 0
Exit policy max trials: 50
Batch size: 512
Output directory: /tmp/output
...
2022-06-28 10:25:58 [INFO] |******Mixed Precision Statistics*****|
2022-06-28 10:25:58 [INFO] +-----------------+----------+--------+
2022-06-28 10:25:58 [INFO] |     Op Type     |  Total   |  INT8  |
2022-06-28 10:25:58 [INFO] +-----------------+----------+--------+
2022-06-28 10:25:58 [INFO] |      Conv2D     |    53    |   53   |
2022-06-28 10:25:58 [INFO] |      MatMul     |    1     |   1    |
2022-06-28 10:25:58 [INFO] |     MaxPool     |    4     |   4    |
2022-06-28 10:25:58 [INFO] |    QuantizeV2   |    5     |   5    |
2022-06-28 10:25:58 [INFO] |    Dequantize   |    4     |   4    |
2022-06-28 10:25:58 [INFO] +-----------------+----------+--------+
2022-06-28 10:25:58 [INFO] Pass quantize model elapsed time: 32164.27 ms
2022-06-28 10:25:58 [INFO] Start to evaluate the TensorFlow model.
2022-06-28 10:26:12 [INFO] Model inference elapsed time: 13921.64 ms
2022-06-28 10:26:12 [INFO] Tune 1 result is: [Accuracy (int8|fp32): 0.9008|0.9022, Duration (seconds) (int8|fp32): 13.9226|17.3321], Best tune result is: [Accuracy: 0.9008, Duration (seconds): 13.9226]
2022-06-28 10:26:12 [INFO] |**********************Tune Result Statistics**********************|
2022-06-28 10:26:12 [INFO] +--------------------+----------+---------------+------------------+
2022-06-28 10:26:12 [INFO] |     Info Type      | Baseline | Tune 1 result | Best tune result |
2022-06-28 10:26:12 [INFO] +--------------------+----------+---------------+------------------+
2022-06-28 10:26:12 [INFO] |      Accuracy      | 0.9022   |    0.9008     |     0.9008       |
2022-06-28 10:26:12 [INFO] | Duration (seconds) | 17.3321  |    13.9226    |     13.9226      |
2022-06-28 10:26:12 [INFO] +--------------------+----------+---------------+------------------+
2022-06-28 10:26:12 [INFO] Save tuning history to /tmp/output/nc_workspace/./history.snapshot.
2022-06-28 10:26:12 [INFO] Specified timeout or max trials is reached! Found a quantized model which meet accuracy goal. Exit.
..
INFO:tensorflow:SavedModel written to: /tmp/output/quantized/resnet_v1_50/1/saved_model.pb
2022-06-28 10:26:13 [INFO] SavedModel written to: /tmp/output/quantized/resnet_v1_50/1/saved_model.pb
2022-06-28 10:26:13 [INFO] Save quantized model to /tmp/output/quantized/resnet_v1_50/1
```

## Benchmark the Quantized Model

The `tlt benchmark` command is used again, but this time the `model-dir` should point
to the quantized model directory.
Make sure to specify your own file paths for `model-dir` and `dataset-dir`. You can then compare
the performance of the full precision model to the quantized model.
```
tlt benchmark --model-dir /tmp/output/quantized/resnet_v1_50/1 --dataset-dir ${DATASET_DIR} --batch-size 512
```
```
Model directory: /tmp/output/quantized/resnet_v1_50/1
Dataset directory: /tmp/dataset/flower_photos
Benchmarking mode: performance
Batch size: 512
Model name: resnet_v1_50
Framework: tensorflow
...
performance mode benchmark result:
2022-06-28 10:28:33 [INFO] Batch size = 512
2022-06-28 10:28:33 [INFO] Latency: 0.946 ms
2022-06-28 10:28:33 [INFO] Throughput: 1056.940 images/sec
```

## Perform Graph Optimization on the Trained Model

Alternatively, the [Intel Neural Compressor](https://intel.github.io/neural-compressor) can be used to optimize
the full precision graph. Make sure to specify your own file paths for `model-dir` and `output-dir`.
Note that graph optimization is also done as part of the quantization flow, so there is no need to call
`tlt optimize` on a quantized model.
```
tlt optimize --model-dir /tmp/output/resnet_v1_50/1 --output-dir /tmp/output
```
```
Model directory: /tmp/output/resnet_v1_50/1
Model name: resnet_v1_50
Output directory: /tmp/output
Framework: tensorflow
Starting graph optimization
...
2022-06-28 13:50:01 [INFO] Graph optimization is done.
...
2022-06-28 13:51:21 [INFO] SavedModel written to: /tmp/output/optimized/resnet_v1_50/1/saved_model.pb
```

## Addititional CLI Examples

More CLI examples can be found here:
* [Image classification examples](/examples/cli/image_classification.md)
* [Text classification examples](/examples/cli/text_classification.md)