4 Replies Latest reply on Feb 18, 2018 10:09 PM by Intel Corporation

    Jupyter Notebook compute power

    DivyanshJha

      There is a hell lot of problems with your jupyter notebook platform.

      1. Very low compute power. Most of the AI work that most researchers do is an interactive process. I cannot submit scripts like you say because I need to monitor stuff. I can only submit scripts after a long interactive process. So your script submission is almost no use to me.

      2. 4H Jupyter notebook time. I don't see a point of limiting the running time of jupyter notebooks.

      3. After I complete two 4h sessions it doesn't let me connect again. Now it's been over an hour and its not letting me connect. I am writing this post in quite a lot of frustration.

      5. SSH into your server takes a minute or two whereas on other servers like Amazon or GCP it happens in seconds.

      6. If intel seriously want to some serious advancement in AI it has to spend some on this platform because currently it is a big pain

       

      Images below

       

      Notebook stuck here

       

       

       

      I logged out and logged back in then this happened

        • 1. Re: Jupyter Notebook compute power
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi DivyanshJha,

          1. We will raise your concern on the compute resource limitation on Jupyter notebook with the concerned team. Meanwhile, for the problem regarding the interactive process, I shall recommend a few options:

          • Login to the compute node using qsub –I and you can run your commands in an interactive mode.
          • You can also make use of screen to make your session interactive.
          • In case you submit the scripts, you can still monitor your real time logs of your job using qpeek command.
          2. The 4hrs Jupyter notebook time is to ensure a fair utilization of resources by all the DevCloud users. Also, connection issues could be due to many users trying to access the server. We shall raise this concern to the respective team and check if the session time could be increased.
          3. SSH into our servers takes only a few seconds for us. If you could share the steps on how you login to our servers, we could check on this and get back to you.
          4. Logging out and logging in does not throw any error. As mentioned, it could be because many users trying to access the system.

           
          Regards,
          Sandhiya

          • 2. Re: Jupyter Notebook compute power
            Intel Corporation
            This message was posted on behalf of Intel Corporation

            Hi DivyanshJha,
            Please update a few details given below to understand your requirement:

            1. Are you using notebook that requires more memory? If yes, how much? Note: currently we have a memory limit of 48Gb.
            2. Did you try out the options provided above using the ssh session? In case you find any issue, please send a screenshot/log so that we can debug and customize.
            Regards,
            Sandhiya
            • 3. Re: Jupyter Notebook compute power
              DivyanshJha

              No, I just use it to train small machine learning models and also for setting up my code so that I can run it on aws . I use devcloud to check whether my code is having errors or not so that i spend less time on aws.

               

              I have to commit my changes as well in a well documented format. So i doubt if those other options are that much helpful.

               

              TLDR

              Increase compute power and time on jupyter notebooks and bandwidth as well. So that no developer faces such problems.

              • 4. Re: Jupyter Notebook compute power
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hi Divyansh,

                Are you still facing this issue? Do let us know if we can go ahead and close this case.

                Reach out to us for any further assistance.

                Thanks & Regards,
                Sandhiya