|
|
/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated |
|
|
and will be removed in future. Use torchrun. |
|
|
Note that |
|
|
If your script expects ` |
|
|
change it to read from `os.environ['LOCAL_RANK']` instead. See |
|
|
https://pytorch.org/docs/stable/distributed.html#launch-utility for |
|
|
further instructions |
|
|
|
|
|
warnings.warn( |
|
|
WARNING:torch.distributed.run: |
|
|
***************************************** |
|
|
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. |
|
|
***************************************** |
|
|
/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory |
|
|
/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python: can't open file 'wav2vec2_fine_tune.py': [Errno 2] No such file or directory |
|
|
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 2) local_rank: 0 (pid: 7680) of binary: /home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/bin/python |
|
|
Traceback (most recent call last): |
|
|
File "/home/itai/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 194, in _run_module_as_main |
|
|
return _run_code(code, main_globals, None, |
|
|
File "/home/itai/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 87, in _run_code |
|
|
exec(code, run_globals) |
|
|
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module> |
|
|
main() |
|
|
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main |
|
|
launch(args) |
|
|
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch |
|
|
run(args) |
|
|
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run |
|
|
elastic_launch( |
|
|
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__ |
|
|
return launch_agent(self._config, self._entrypoint, list(args)) |
|
|
File "/home/itai/.pyenv/versions/venv-nlu_pipeline-wav2vec/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent |
|
|
raise ChildFailedError( |
|
|
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: |
|
|
============================================================ |
|
|
wav2vec2_fine_tune.py FAILED |
|
|
|
|
|
Failures: |
|
|
[1]: |
|
|
time : 2022-04-13_09:16:59 |
|
|
host : itai-gpu-dev1.internal.cloudapp.net |
|
|
rank : 1 (local_rank: 1) |
|
|
exitcode : 2 (pid: 7681) |
|
|
error_file: <N/A> |
|
|
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
|
|
|
|
|
Root Cause (first observed failure): |
|
|
[0]: |
|
|
time : 2022-04-13_09:16:59 |
|
|
host : itai-gpu-dev1.internal.cloudapp.net |
|
|
rank : 0 (local_rank: 0) |
|
|
exitcode : 2 (pid: 7680) |
|
|
error_file: <N/A> |
|
|
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html |
|
|
============================================================ |
|
|
|
