1 2 Previous Next 19 Replies Latest reply on Dec 7, 2017 4:07 AM by Ctrlaltlab

    Space limitation for imagenet dataset

    Ctrlaltlab

      Hi All,

      i tried to use caffe framework on node c009 with the sample imagenet that come with the source.

      I had download the dataset ILSVRC2012 for the train, val and test in .tar format.

      The problem is about memory limitation when I try to decompress the files.

      How I can go forward to do that?

       

      Thanks in advanced.

        • 1. Re: Space limitation for imagenet dataset
          HemanthK

          Hi,

          The quota for your home folder is 200 GB. Home folder is NFS-shared between the login node and the compute nodes. If your data fits within that quota, you should be good to go.

           

          Do not use /tmp.


          Also Some machine learning datasets can be found in /local/

          Best
          Hemanth

          • 2. Re: Space limitation for imagenet dataset
            Ctrlaltlab

            Hi Hemanth,

             

            i understood the 200GB limitation, but this means that I can't try the imagenet example of caffe..

            because the dataset need more space..

             

            About /local/ Folder for me not exist. Is this my fault?

             

            I'm on c009.

             

            Thanks in advanced.

             

             

            • 3. Re: Space limitation for imagenet dataset
              HemanthK

              Hi,

              /local directory should be visible from all nodes. Let us know your user id on c009 and we can check and remediate.

              Alternate solution to the dataset size : we can have dataset ILSVRC2012 decompressed and made available in /local

              Thanks

              Hemanth

              • 4. Re: Space limitation for imagenet dataset
                Ctrlaltlab

                Hi Hemanth,

                this is my user id u6985 on c009.

                 

                Thanks for your time.

                C.

                • 5. Re: Space limitation for imagenet dataset
                  Anju_Paul

                  Hi,

                   

                  Some of the commonly used data-sets are available in the folder /data

                  Could you please check if that helps you?

                   

                  Thanks,

                  Anju

                  • 6. Re: Space limitation for imagenet dataset
                    Ctrlaltlab

                    Hi Anju,

                     

                    the Datasets are available in /data.

                     

                    Thanks for your support..!

                    • 7. Re: Space limitation for imagenet dataset
                      Ctrlaltlab

                      Hi to all,

                      to understand the cluster best practices, explain the state of the art of my problem.

                       

                      My goal is to set the imaginet example with caffe.

                      Now I have access to the data set in / data / imagenet_2012.

                       

                      I need to resize all images to size 256x256, as suggestions from caffe examples.

                       

                      Can I modify this dataset?

                      What can I do in this case without coming into conflict with other people's claims?

                       

                      Thanks in advanced.

                       

                      C.

                      • 8. Re: Space limitation for imagenet dataset
                        Ratheesh_Intel

                         

                        Hi Ctrlaltlab,

                         

                             Intel caffe enables us to input the data to the network in different ways, we can directly pass the path of the image to the network or  we can create LMDB then pass the path of the lmdb to the network. we recommend to use LMDB. LMDB is designed for faster fetching of data and it is stored in uncompressed format, It becomes very easy for the machine to read the data and directly pass them for processing.

                         

                        For your case, you don't want to explicitly resize the image since the script for lmdb creation is already doing the same. we can mention the height and width of the images to be converted while you run the script which creates the LMBD files in the respective folders. 

                         

                         

                        Thanks & Regards

                        Ratheesh A

                         

                        1 of 1 people found this helpful
                        • 9. Re: Space limitation for imagenet dataset
                          Ctrlaltlab

                          Hi Ratheesh A.,

                          thanks for explanation..it was very useful.

                           

                          You could also tell me how to compile from cluster examples cpp without installing caffe from source?

                          More in detail, can I compile example cpp_classification with the caffe intel compiler?

                           

                          Or I need to install caffe from source to do that?

                           

                          if I run command "which caffe" give me /glob/intel-python/python2/bin/caffe on the node c009.

                          Is this release of caffe installed on Colfax compiled only for python?

                           

                          Thanks for your time..!

                          • 10. Re: Space limitation for imagenet dataset
                            Ratheesh_Intel

                            Hi,

                             

                            For compilation you can use gcc (gcc <filename.cpp>)

                             

                            For testing your pycaffe installations,

                             

                            1. Open your bash terminal, and type python

                            2. import caffe and observe if the statement executes successfully

                             

                            Your caffe installation seems to be perfect.

                             

                            Thanks & Regards

                            Ratheesh A

                            • 11. Re: Space limitation for imagenet dataset
                              Ctrlaltlab

                              Hi Ratheesh A,

                              I'm confused with your answer..XD

                              I try to be more concise:

                               

                              I tried to compile the tools/imageset.cpp, without compile the frameworks caffe.

                              gcc -I/home/u6985/caffe/include/ classification.cpp -o classification

                              In file included from /home/u6985/caffe/include/caffe/common.hpp:56:0,

                                               from /home/u6985/caffe/include/caffe/blob.hpp:45,

                                               from /home/u6985/caffe/include/caffe/caffe.hpp:44,

                                               from classification.cpp:38:

                              /home/u6985/caffe/include/caffe/util/device_alternate.hpp:71:23: fatal error: cublas_v2.h: No such file or directory

                              #include <cublas_v2.h>

                                                     ^

                              compilation terminated.

                               

                                   I can do that?

                                   if yes, where I need to found the include path and library path for the gcc command?

                               

                               

                              Thanks in advanced.

                              • 12. Re: Space limitation for imagenet dataset
                                Ratheesh_Intel

                                Hi Ctrlaltlab,

                                 

                                Are you trying to install Caffe on your own? Is there a specific purpose as to why you are doing it?

                                The environment comes pre-installed with Caffe and is ready to use.

                                 

                                If you are making changes to the /tools/imageset.cpp, please note that the C++ components in the Caffe gets built while we issue the MAKE command and all the libraries and the paths are configured in the Makefile.config for successfully compiling the components. In this case, you are trying to use gcc and compile specific components, but it still require lot of dependent libraries and hence you will have to specify it in your LD_LIBRARY_PATH. But this is really cumbersome and  I would suggests you to do a 'Make' again to rebuild your components inside caffe

                                 

                                Thanks

                                Ratheesh

                                • 13. Re: Space limitation for imagenet dataset
                                  Ctrlaltlab

                                  Hi Rathesssh,

                                  thanks for your answer .. I can't said is simple to understand

                                   

                                  You said :Are you trying to install Caffe on your own? Is there a specific purpose as to why you are doing it?

                                  Really, I wanna skip this step, I downloaded the source for the example. I have some issue to use it.

                                   

                                  As you said : The environment comes pre-installed with Caffe and is ready to use.

                                       Where I can found the include and the library for caffe on the cluster?

                                   

                                  When you speak about make change of /tools/imageset.cpp, which caffe .cpp file do you speak about ?  From the source in my home_dir?

                                   

                                   

                                  Like always...Thanks in advanced.

                                   

                                   

                                   

                                   

                                   

                                   

                                   

                                      

                                   

                                      

                                   

                                   

                                  • 14. Re: Space limitation for imagenet dataset
                                    Ratheesh_Intel

                                    Hi Ctrlaltlab,

                                     

                                    As I mentioned earlier Caffe is already being set up and available in your environment.   Please login to your environment. For training your model please create all the necessary files such as solver.prototxt, train.prototxt and test.prototxt and you can simply run 'caffe train -solver <path to your solver file>'. caffe command is available in every path of your environment.

                                        

                                    please find the below image. Iam inside my environment and I just tried to execute caffe train command, it accepted the command, but i did not gave the path to the solver file that is what the error shows. but caffe is all set there.

                                     

                                     

                                     

                                    Will sent you step by step procedure to execute a sample code as soon as possible.

                                     

                                    Thanks & Regards

                                    Ratheesh A

                                    1 2 Previous Next