efficientdet_comparison/readme.md

# Performance Benchmark of Quantized Detection Model

This directory is built for comparison of our "quantized / quantization aware trained" detection algorithm to one of the SOTA compact detection algorithms, EfficientDet-d0, which has comparable complexity and structure with our quantized model.

Our person tracking algorithm uses MobileNet-v2 as backbone mechanism and combines it with 2 SSD heads using total of 9 anchor boxes. Overall model consists of 60 convolution layers. 

We quantized the layers of this model and applied "quantization aware training" methods to recover its accuracy drop due to quantization of layers and output clamping. We have re-scaled and center cropped the images in [Wider Person Dataset](https://competitions.codalab.org/competitions/20132#learn_the_details), also we resized its annotations and converted in to COCO annotation format to use them in our training/evaluation tasks. Then we applied smart training approaches which consider the effects of quantization and output clamping of the layers during optimization, which we call "quantization aware training".

Our main motivation of quantizing networks and applying quantization aware training methods is to reduce the overall network size, inference time and training effort while keeping accuracy drop in an acceptable level. We aim to develop quantized compact detection algorithms executable on low power and low cost accelerator chips.

## Dependencies
* [PyTorch](https://github.com/pytorch/pytorch)
* [Torchvision](https://github.com/pytorch/vision)
* [Pycocotools](https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools)
* [webcolors](https://pypi.org/project/webcolors/)
* [PyYAML](https://github.com/yaml/pyyaml)

## Evaluating EfficientDet with Wider Person Validation Dataset
In this section, steps to reproduce the evaluation of EfficientDet model from [Yet-Another-EfficientDet-Pytorch Repository](https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch.git) with d0 coefficients is explained. For evaluation, aforementioned Wider Person Validation Dataset in COCO format is used. 

### 1. Clone EfficientDet to Your Local
Open a terminal and go to directory in your local where you want to clone , then type:
```bash
git clone --depth 1 https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch
```

### 2. Prepare EfficientDet-d0 Coefficients
* Go to main directory
    ```bash
    cd Yet-Another-EfficientDet-Pytorch/
    ```
* Create weights folder
    ```bash
    mkdir weights
    ```
* Download EfficientDet d0 coefficients 
    ```bash
    wget https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/releases/download/1.0/efficientdet-d0.pth -O weights/efficientdet-d0.pth
    ```

### 3. Prepare Wider Person Dataset
* Download original Wider Person Dataset
    * Sign up [Codalab](https://competitions.codalab.org/) and  participate to [WIDER Face & Person Challenge 2019](https://competitions.codalab.org/competitions/20132)
    * Under "Participate" tab click "Train & Validation Data in Google Drive" and download
        * val_data.tar.gz
        * Annotations/val_bbox.txt
    * Extract val_data.tar.gz as val_data and move val_data folder under ./data/original_wider/val_data
    * Move "val_bbox.txt" under ./data/original_wider/

* Move our "wider2coco.py" script in "efficientdet_comparison" folder to main folder of your local "Yet-Another-EfficientDet-Pytorch" repository. Following code will produce resized images and annotations.
    ```bash
    python wider2coco.py -ip ./data/original_wider/val_data -af ./data/original_wider/val_bbox.txt 
    ```
* Script will automatically convert Wider Dataset in to COCO format and create following repository structure:

        ./Yet-Another-EfficientDet-Pytorch/datasets/wider/val
             image001.jpg
             image002.jpg
             ...
        ./Yet-Another-EfficientDet-Pytorch/datasets/wider/annotations
            instances_val.json
    
   

### 4. Manually Set Project's Specific Parameters

* Create a yml file "wider.yml" under "projects"
    ```bash
    touch projects/wider.yml
    ```

 * Copy following content in to "wider.yml" file
 
       project_name: wider
       train_set: train
       val_set: val
       num_gpus: 1  # 0 means using cpu, 1-N means using gpus 
        
       # Wider validation dataset mean and std in RGB order
       mean: [0.416, 0.413, 0.406]
       std: [0.308, 0.306, 0.310]
        
       # this is coco anchors, change it if necessary
       anchors_scales: '[2 ** 0, 2 ** (1.0 / 3.0), 2 ** (2.0 / 3.0)]'
       anchors_ratios: '[(1.0, 1.0), (1.4, 0.7), (0.7, 1.4)]'
        
       # objects from all labels from your dataset with the order from your annotations.
       # its index must match your dataset's category_id.
       # category_id is one_indexed,
       # for example, index of 'car' here is 2, while category_id of is 3
       obj_list: ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', '', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', '', 'backpack', 'umbrella', '', '', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', '', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', '', 'dining table', '', '', 'toilet', '', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', '', 'book', 'clock', 'vase', 'scissors','teddy bear', 'hair drier', 'toothbrush']

### 5. Evaluate EfficientDet model performance
* Setup "person only evaluation"
    * Open "coco_eval.py" under <parent directory>/Yet-Another-EfficientDet-Pytorch
    * Paste following code after line 132, "coco_eval.params.imgIds = image_ids", to evaluate mAP results only for person category
        ```python
        coco_eval.params.catIds = 1
        ```

* For evaluation on cuda enabled platform 
    ```bash
    python coco_eval.py -p wider -c 0 -w ./weights/efficientdet-d0.pth
    ```

* For evaluation on cuda disabled platform 
    ```bash
    python coco_eval.py -p wider -c 0 -w ./weights/efficientdet-d0.pth --cuda False
    ```


## Evaluating Our Quantized MobilenetSSDLite model with Wider Person Validation Dataset

### 1. Clone Quantized Mobilenet Model to Your Local
Open a terminal and go to directory in your local where you want to clone our [Quantization Aware Training - Person Tracking](https://github.com/sai-tr/persontracking_qat.git) repository, then type:
```bash
git clone --depth 1 https://github.com/sai-tr/persontracking_qat.git
```

### 2. Prepare Wider Person Dataset
* Download original Wider Person Dataset
    * Sign up [Codalab](https://competitions.codalab.org/) and  participate to [WIDER Face & Person Challenge 2019](https://competitions.codalab.org/competitions/20132)
    * Under "Participate" tab click "Train & Validation Data in Google Drive" and download
        * val_data.tar.gz
        * Annotations/val_bbox.txt
    * Extract val_data.tar.gz as val_data and move val_data folder under ./data/original_wider/val_data
    * Move "val_bbox.txt" under ./data/original_wider/

* Move our "wider2coco.py" script in "efficientdet_comparison" folder to main folder of your local "persontracking_qat" repository. Following code will produce resized images and annotations.
    ```bash
    python wider2coco.py -ip ./data/original_wider/val_data -af ./data/original_wider/val_bbox.txt 
    ```
* Script will automatically convert Wider Dataset in to COCO format and create following repository structure:

        ./persontracking_qat/datasets/wider/val
             image001.jpg
             image002.jpg
             ...
        ./persontracking_qat/datasets/wider/annotations
            instances_val.json

### 3. Evaluate Quantized Mobilenet Model Performance
Note that model mode should match with the loaded model parameter dictionary. Selectable model modes are: 
* Full Precision Unconstrained(fpt_unc): All layers are in full precision and no output clamping
* Full Precision Constrained(fpt): All layers are in full precision and layer output are clamped to +-1 
* Quantized(qat): All layers are quantized layer outputs are clamped to +-1


* Move our "coco_eval.py" script in "efficientdet_comparison" folder to "persontracking_qat" folder and use following command for evaluation:
    ```bash
    python coco_eval.py -m qat -dp ./datasets/wider/val -ap ./datasets/wider/annotations/all_val_prep.json -wp ./efficientdet_comparison/training_experiment_best.pth.tar
    ```
    Note that: Code evaluates quantized model with weights "training_experiment_best.pth.tar", using images and annotations in paths  "./datasets/wider/val" "./datasets/wider/annotations/instances_val.json" respectively.

## mAP Comparisons
### EfficientDet-d0
    ### Wider Validation Dataset mAP scores ###
    Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.292
    Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.543
    Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.275
    Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.109
    Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.409
    Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.532
    Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.106
    Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.369
    Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.435
    Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.270
    Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.546
    Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.678
    

### Quantized Mobilenet
    ### Wider Validation Dataset mAP scores ###
    Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.281
    Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.457
    Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.310
    Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.075
    Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.406
    Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.582
    Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.107
    Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.324
    Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.331
    Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.110
    Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.481
    Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.637