10 Replies Latest reply on May 23, 2018 11:08 PM by Intel Corporation

    Cannot run tensorboard on DevCloud

    kshitizrimal

      When I tried to run tensorboard on devcloud it shows error

       

      Traceback (most recent call last):

        File "/glob/intel-python/versions/2018u2/intelpython2/bin/tensorboard", line 4, in <module>

          import tensorflow.tensorboard.tensorboard

      ImportError: No module named tensorboard.tensorboard

       

      I tried updating tensorflow, re-installing tensorboard but nothing works. I tried running from node or directly nothing works.

        • 1. Re: Cannot run tensorboard on DevCloud
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi
          Thanks for reaching us. will look into it and get back to you
          Thanks

          • 2. Re: Cannot run tensorboard on DevCloud
            Intel Corporation
            This message was posted on behalf of Intel Corporation

            Hi 
            Can you please create conda environment and try import tensorboard
            1)conda create -n tensorboard_env intepython3_core python=3
            2)source activate tensorboard_env
            3) conda install tensorboard
            4) conda install tensorflow
            now try importing tensorboard. Please find the attachment.

            • 3. Re: Cannot run tensorboard on DevCloud
              kshitizrimal

              Hi, how can I run that by importing? I was running that using tensorboard --log-dir=<dir> command directly from command line. When I tried the above instructions, it still displays same error while running from command line.

              • 4. Re: Cannot run tensorboard on DevCloud
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hi 
                Can you please follow below steps to run tensorboard
                1)Add a tunnel in ssh putty with unique source port and destination port as 6006.Please find attachment(tensorboard1.jpg) to know more information more about how to add  source port and destination port
                2)qsub -I
                3)conda create -n tensorboard_env python=3
                4)source activate tensorboard_env
                5)conda install tensorflow
                6)conda install tensorboard
                7)Run plink command on command prompt as below
                 7a)plink -ssh -L 60255:localhost:6006 DevCloud ssh -L 6006:localhost:6006 c009-n004
                8)python -m tensorboard --logdir=/path/to/logs &
                9)open below link in browser 
                http://localhost:60255 

                In all the above steps i have considered 60255 as sample source port which is not fixed one but destination port should be 6006

                • 5. Re: Cannot run tensorboard on DevCloud
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  Hi 
                  Small typo in the previous post. i am putting all the steps again
                  1)Add a tunnel in ssh putty with unique source port and destination port as 6006.Please find attachment(tensorboard1.jpg) to know more information more about how to add source port and destination port
                  2)qsub -I
                  3)conda create -n tensorboard_env python=3
                  4)source activate tensorboard_env
                  5)conda install tensorflow
                  6)conda install tensorboard
                  7)Run plink command on command prompt as below
                  7a)plink -ssh -L 60255:localhost:6006 DevCloud ssh -L 6006:localhost:6006 c009-n004
                  8)python -m tensorboard.main --logdir=/path/to/logs &
                  9)open below link in browser 
                  http://localhost:60255 
                  And 60255 is my local port. You can use any number as local port in the range of (49152,65535)
                  Thanks
                   

                  • 6. Re: Cannot run tensorboard on DevCloud
                    kshitizrimal

                    Thank you so much for the reply. My system is macOs and I use iTerm for ssh. In given example you have attached screenshot of a windows machine. Is there any guide for mac users?

                     

                    Without the forwarding and tunneling when i tried other commands (to create environment and launch tensorboard) it works. So if there is any way to connect localhost of my local machine to that of devcloud that would be great.I tried this command to link my localhost with that of devcloud, but it doesn't work.

                     

                    ssh -L 6006:localhost:6006 <loginHost> ssh -L 6006:localhost:6006 <node name>

                     

                    for loginHost i used "devcloud", which is the alias for the devcloud cluster in my machine and for node name is used, c009-n004.

                     

                    ssh -L 6006:localhost:6006 devcloud ssh -L 6006:localhost:6006 c009-n004

                    • 7. Re: Cannot run tensorboard on DevCloud
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Hi 
                      Please follow below steps to open tensorboard on MAC/Linux systems on new SSH prompt
                      1)ssh –L 6006:localhost:6006 XXXX (XXXX is the name that we will provide while we are trying to access the cluster using a Secure Shell (SSH) client i.e in  ~/.ssh/config file the line is "Host XXXX")
                      2)conda create -n tensorboard_env intelpython3_core python=3
                      3)source activate tensorbord_env
                      4)python -m tensorboard --logdir=/path/to/logs &
                      5)open your local browser with below link
                      http://localhost:6006
                       

                      • 8. Re: Cannot run tensorboard on DevCloud
                        kshitizrimal

                        Hi, when I tried your latest instructions and when I tried to launch tensorboard from devcloud, it launches the tensorboard but when i try to open from my browser with locahost:6006, it says this:

                         

                        channel 3: open failed: connect failed: Connection refused

                         

                        It looks like i successfully ssh with localhost but when I try to open from my browser after launching tensorboard it gives above message and tensorboard doesn't open in my browser. However it is running on my devcloud.

                         

                        Here are the screenshots for it:

                        Screen Shot 2018-05-20 at 11.13.28 AM.png

                         

                        Screen Shot 2018-05-20 at 11.13.39 AM.png

                        • 9. Re: Cannot run tensorboard on DevCloud
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          Hi Kshitiz,
                          As per the screen shots, I have observed that it was running on compute node and conda environment is not activated.
                          Can you please re-run tensorboard on login node after the activation of your conda environment. Please find the attachment to have more clarity about steps.
                          And while loading tensorboard UI in browser, takes an awfully long time. In case of any Connection time out error, Please refresh the page

                          Thanks
                           

                          • 10. Re: Cannot run tensorboard on DevCloud
                            Intel Corporation
                            This message was posted on behalf of Intel Corporation

                            Hi kshitiz,
                            We didn't hear from you.Hope it has answered your queries.We are closing this thread.If you have any other issue,Please raise a new thread.
                            Thanks