Please confirm if you wanted to see the real time logs of the running job in jupyter notebook (hub).
Also, could you please let us know if you run the python code directly or use the qsub utility with jupyter notebook.
Kindly share more details on how you submit the job in jupyter notebook.
Yes I want to see real-time logs it could be from the terminal or a jupyter notebook.
I used `qsub` utility from the terminal:
Here my steps:
1. I used this instruction to open the terminal: Using Jupyter Notebook* Terminal Console | Intel® Software
- Created conda environment.
- activated environment.
- Installed some libraries
- created file "myjob" with this lines:
#PBS -l nodes=1
echo Starting calculation
source activate iMaterialist
echo End of calculation
- executed `qsub myjob` from the terminal
- started training job with number xxxxx.c00x
- It ran for certain hours.
3. The job stopped suddenly and I don't get any information about the process.
4. My model saves a basic log and it showed me it was on the epoch 14/189 way far to the end.
I am sorry I think I made another question related here:
To view the real time logs of a running job, we use qpeek <JOB_ID> & qpeek –e <JOB_ID>.
Unfortunately, the qpeek command does not work in terminal of jupyter notebook hub now.
Hence we would suggest you three options:
- Terminal that you get through Jupyter notebook(hub) is actually that of a compute node and hence you could directly run the commands there, as you do in a local machine. i.e. no need to use qsub utility at all. This will give the logs just as in a local machine. The drawback of this is that, the terminal when inactive, could lose connection and hence you might stop seeing the logs beyond a point of time. Secondly, since there is 4 hour session time limit with Jupyter notebook(hub), you could view the logs only for that long.
- Submit the job using the jupyter notebook terminal. Save the job id. Login using ssh with the help of putty or linux terminal (Steps are available in the link provided with the welcome mail of DevCloud). Run the commands, qpeek <JOB_ID> and qpeek –e <JOB_ID> in the ssh terminal.
- Since you are already familiar with the basic linux commands and qsub utility, all the steps that you do currently in Jupyter notebook terminal could be done in the ssh terminal. This option will help to avoid switching between jupyter notebook & ssh terminal.