Spaces:
Configuration error
Configuration error
File size: 2,380 Bytes
a01ef8c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
# Downloader
An easy-to-use, unified tool for downloading and managing AI datasets and models.
## Datasets
### Supported Catalogs & File Types
| Source | Info |
|----------|-----------|
| TensorFlow Datasets | [https://www.tensorflow.org/datasets](https://www.tensorflow.org/datasets) |
| Torchvision | [https://pytorch.org/vision/stable/datasets.html](https://pytorch.org/vision/stable/datasets.html) |
| Hugging Face | [https://huggingface.co/docs/datasets/index](https://huggingface.co/docs/datasets/index) |
| Generic Web URL | Publicly downloadable files: `.zip`, `.gz`, `.bz2`, `.txt`, `.csv`, `.png`, `.jpg`, etc. |
### Usage
Dataset catalog example:
```
from downloader.datasets import DataDownloader
downloader = DataDownloader('tf_flowers', dataset_dir='/home/user/datasets', catalog='tensorflow_datasets')
downloader.download(split='train')
```
URL example:
```
from downloader.datasets import DataDownloader
downloader = DataDownloader('my_dataset', dataset_dir='/home/user/datasets', url='http://<domain>/<filename>.zip')
downloader.download()
```
## Models
### Supported Model Hubs
| Source | Info |
|----------|-----------|
| TensorFlow Hub | [https://www.tensorflow.org/hub](https://www.tensorflow.org/hub) |
| Torchvision | [https://pytorch.org/vision/stable/models.html](https://pytorch.org/vision/stable/models.html) |
| Hugging Face | [https://huggingface.co/models](https://huggingface.co/models) (AutoModelForSequenceClassification or TFBertModel types) |
### Usage
Example:
```
from downloader.models import ModelDownloader
# Hugging Face
downloader = ModelDownloader('bert-large-uncased', hub='hugging_face', num_labels=2)
downloader.download()
# Torchvision
downloader = ModelDownloader('resnet34', hub='torchvision')
downloader.download()
```
## Build and Install
To install the downloader, follow [The setup instructions for Intel Transfer Learning Tool](/README.md#build-and-install). The downloader is currently
packaged alongside the Intel Transfer Learning Tool and uses its requirements.txt files, but the tools can be separated at some future time. The
downloader's dependencies are tracked in [requirements.txt](requirements.txt).
## Testing
With an activated environment that has the dependencies for the downloader and `pytest` in it, run this command from
the root repository directory:
```
py.test -s downloader/tests
```
|