3 Replies Latest reply on May 17, 2018 12:16 AM by Intel Corporation

    An "Segmentation Fault" Error of Intel Caffe on Intel AI DevCloud

    shf19961008

      We run intel-caffe(version 1.1.0) on Intel AI DevCloud but got an "Segmentation fault" error, however, the same model and data run successfully on my own laptop with bvlc-caffe(version 1.0.0).

      The model and data:

      https://drive.google.com/open?id=1WzfL1Ujow2RvtwS4oQx2Hhr2I8zEewVH

      all files are placed under home directory.

      Anyone can help?

      I0515 01:55:22.797646 231762 caffe.cpp:285] Use CPU.
      I0515 01:55:22.838340 231762 solver.cpp:107] Initializing solver from parameters:
      base_lr: 0.01
      display: 10
      max_iter: 100
      lr_policy: "fixed"
      weight_decay: 0.0005
      snapshot: 20
      snapshot_prefix: "/home/u11804/snapshot"
      solver_mode: CPU
      debug_info: true
      net: "sketch_net_.prototxt"
      train_state {
        level: 0
        stage: ""
      }
      I0515 01:55:22.840106 231762 solver.cpp:153] Creating training net from net file: sketch_net_.prototxt
      I0515 01:55:22.861557 231762 cpu_info.cpp:453] Processor speed [MHz]: 3400
      I0515 01:55:22.861600 231762 cpu_info.cpp:456] Total number of sockets: 2
      I0515 01:55:22.861608 231762 cpu_info.cpp:459] Total number of CPU cores: 12
      I0515 01:55:22.861614 231762 cpu_info.cpp:462] Total number of processors: 24
      I0515 01:55:22.861620 231762 cpu_info.cpp:465] GPU is used: no
      I0515 01:55:22.861626 231762 cpu_info.cpp:468] OpenMP environmental variables are specified: no
      I0515 01:55:22.861632 231762 cpu_info.cpp:471] OpenMP thread bind allowed: yes
      I0515 01:55:22.861639 231762 cpu_info.cpp:474] Number of OpenMP threads: 12
      I0515 01:55:22.861763 231762 net.cpp:1052] The NetState phase (0) differed from the phase (1) specified by a rule in layer data
      I0515 01:55:22.861804 231762 net.cpp:1052] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
      I0515 01:55:22.862501 231762 net.cpp:207] Initializing net from parameters:
      I0515 01:55:22.862512 231762 net.cpp:208]
      I0515 01:55:22.862963 231762 layer_factory.hpp:114] Creating layer data
      I0515 01:55:22.862990 231762 net.cpp:265] Creating Layer data
      I0515 01:55:22.863001 231762 net.cpp:1238] data -> data
      I0515 01:55:22.863021 231762 net.cpp:1238] data -> labels
      I0515 01:55:22.863039 231762 hdf5_data_layer.cpp:116] Loading list of HDF5 filenames from: sketch_train_dataset_location.txt
      I0515 01:55:22.864107 231762 hdf5_data_layer.cpp:130] Number of HDF5 files: 1
      I0515 01:55:22.906950 231762 hdf5.cpp:69] Datatype class: H5T_FLOAT
      I0515 01:55:25.065345 231762 net.cpp:360] Setting up data
      I0515 01:55:25.065379 231762 net.cpp:367] Top shape: 16 6 256 256 (6291456)
      I0515 01:55:25.065390 231762 net.cpp:367] Top shape: 16 (16)
      I0515 01:55:25.065393 231762 net.cpp:375] Memory required for data: 25165888
      I0515 01:55:25.065408 231762 layer_factory.hpp:114] Creating layer conv1
      I0515 01:55:25.065440 231762 net.cpp:265] Creating Layer conv1
      I0515 01:55:25.065443 231762 net.cpp:1264] conv1 <- data
      I0515 01:55:25.065451 231762 net.cpp:1238] conv1 -> conv1
      I0515 01:55:25.065974 231762 net.cpp:360] Setting up conv1
      I0515 01:55:25.065982 231762 net.cpp:367] Top shape: 16 64 81 81 (6718464)
      I0515 01:55:25.065986 231762 net.cpp:375] Memory required for data: 52039744
      I0515 01:55:25.065996 231762 layer_factory.hpp:114] Creating layer relu1
      I0515 01:55:25.066006 231762 net.cpp:265] Creating Layer relu1
      I0515 01:55:25.066009 231762 net.cpp:1264] relu1 <- conv1
      I0515 01:55:25.066015 231762 net.cpp:1225] relu1 -> conv1 (in-place)
      I0515 01:55:25.066028 231762 net.cpp:360] Setting up relu1
      I0515 01:55:25.066030 231762 net.cpp:367] Top shape: 16 64 81 81 (6718464)
      I0515 01:55:25.066035 231762 net.cpp:375] Memory required for data: 78913600
      I0515 01:55:25.066037 231762 layer_factory.hpp:114] Creating layer pool1
      I0515 01:55:25.066045 231762 net.cpp:265] Creating Layer pool1
      I0515 01:55:25.066048 231762 net.cpp:1264] pool1 <- conv1
      I0515 01:55:25.066052 231762 net.cpp:1238] pool1 -> pool1
      I0515 01:55:25.066071 231762 net.cpp:360] Setting up pool1
      I0515 01:55:25.066073 231762 net.cpp:367] Top shape: 16 64 40 40 (1638400)
      I0515 01:55:25.066077 231762 net.cpp:375] Memory required for data: 85467200
      I0515 01:55:25.066079 231762 layer_factory.hpp:114] Creating layer conv2
      I0515 01:55:25.066087 231762 net.cpp:265] Creating Layer conv2
      I0515 01:55:25.066090 231762 net.cpp:1264] conv2 <- pool1
      I0515 01:55:25.066094 231762 net.cpp:1238] conv2 -> conv2
      I0515 01:55:25.067028 231762 net.cpp:360] Setting up conv2
      I0515 01:55:25.067036 231762 net.cpp:367] Top shape: 16 128 36 36 (2654208)
      I0515 01:55:25.067040 231762 net.cpp:375] Memory required for data: 96084032
      I0515 01:55:25.067054 231762 layer_factory.hpp:114] Creating layer relu2
      I0515 01:55:25.067062 231762 net.cpp:265] Creating Layer relu2
      I0515 01:55:25.067065 231762 net.cpp:1264] relu2 <- conv2
      I0515 01:55:25.067070 231762 net.cpp:1225] relu2 -> conv2 (in-place)
      I0515 01:55:25.067075 231762 net.cpp:360] Setting up relu2
      I0515 01:55:25.067077 231762 net.cpp:367] Top shape: 16 128 36 36 (2654208)
      I0515 01:55:25.067080 231762 net.cpp:375] Memory required for data: 106700864
      I0515 01:55:25.067085 231762 layer_factory.hpp:114] Creating layer pool2
      I0515 01:55:25.067090 231762 net.cpp:265] Creating Layer pool2
      I0515 01:55:25.067093 231762 net.cpp:1264] pool2 <- conv2
      I0515 01:55:25.067097 231762 net.cpp:1238] pool2 -> pool2
      I0515 01:55:25.067104 231762 net.cpp:360] Setting up pool2
      I0515 01:55:25.067106 231762 net.cpp:367] Top shape: 16 128 18 18 (663552)
      I0515 01:55:25.067109 231762 net.cpp:375] Memory required for data: 109355072
      I0515 01:55:25.067111 231762 layer_factory.hpp:114] Creating layer conv3
      I0515 01:55:25.067119 231762 net.cpp:265] Creating Layer conv3
      I0515 01:55:25.067121 231762 net.cpp:1264] conv3 <- pool2
      I0515 01:55:25.067126 231762 net.cpp:1238] conv3 -> conv3
      I0515 01:55:25.068449 231762 net.cpp:360] Setting up conv3
      I0515 01:55:25.068457 231762 net.cpp:367] Top shape: 16 256 18 18 (1327104)
      I0515 01:55:25.068461 231762 net.cpp:375] Memory required for data: 114663488
      I0515 01:55:25.068467 231762 layer_factory.hpp:114] Creating layer relu3
      I0515 01:55:25.068478 231762 net.cpp:265] Creating Layer relu3
      I0515 01:55:25.068480 231762 net.cpp:1264] relu3 <- conv3
      I0515 01:55:25.068485 231762 net.cpp:1225] relu3 -> conv3 (in-place)
      I0515 01:55:25.068490 231762 net.cpp:360] Setting up relu3
      I0515 01:55:25.068491 231762 net.cpp:367] Top shape: 16 256 18 18 (1327104)
      I0515 01:55:25.068495 231762 net.cpp:375] Memory required for data: 119971904
      I0515 01:55:25.068497 231762 layer_factory.hpp:114] Creating layer conv4
      I0515 01:55:25.068505 231762 net.cpp:265] Creating Layer conv4
      I0515 01:55:25.068507 231762 net.cpp:1264] conv4 <- conv3
      I0515 01:55:25.068511 231762 net.cpp:1238] conv4 -> conv4
      I0515 01:55:25.071136 231762 net.cpp:360] Setting up conv4
      I0515 01:55:25.071143 231762 net.cpp:367] Top shape: 16 256 18 18 (1327104)
      I0515 01:55:25.071147 231762 net.cpp:375] Memory required for data: 125280320
      I0515 01:55:25.071152 231762 layer_factory.hpp:114] Creating layer relu4
      I0515 01:55:25.071158 231762 net.cpp:265] Creating Layer relu4
      I0515 01:55:25.071161 231762 net.cpp:1264] relu4 <- conv4
      I0515 01:55:25.071166 231762 net.cpp:1225] relu4 -> conv4 (in-place)
      I0515 01:55:25.071171 231762 net.cpp:360] Setting up relu4
      I0515 01:55:25.071172 231762 net.cpp:367] Top shape: 16 256 18 18 (1327104)
      I0515 01:55:25.071175 231762 net.cpp:375] Memory required for data: 130588736
      I0515 01:55:25.071177 231762 layer_factory.hpp:114] Creating layer conv5
      I0515 01:55:25.071184 231762 net.cpp:265] Creating Layer conv5
      I0515 01:55:25.071187 231762 net.cpp:1264] conv5 <- conv4
      I0515 01:55:25.071192 231762 net.cpp:1238] conv5 -> conv5
      I0515 01:55:25.073843 231762 net.cpp:360] Setting up conv5
      I0515 01:55:25.073851 231762 net.cpp:367] Top shape: 16 256 18 18 (1327104)
      I0515 01:55:25.073855 231762 net.cpp:375] Memory required for data: 135897152
      I0515 01:55:25.073861 231762 layer_factory.hpp:114] Creating layer relu5
      I0515 01:55:25.073868 231762 net.cpp:265] Creating Layer relu5
      I0515 01:55:25.073870 231762 net.cpp:1264] relu5 <- conv5
      I0515 01:55:25.073874 231762 net.cpp:1225] relu5 -> conv5 (in-place)
      I0515 01:55:25.073879 231762 net.cpp:360] Setting up relu5
      I0515 01:55:25.073882 231762 net.cpp:367] Top shape: 16 256 18 18 (1327104)
      I0515 01:55:25.073885 231762 net.cpp:375] Memory required for data: 141205568
      I0515 01:55:25.073889 231762 layer_factory.hpp:114] Creating layer pool5
      I0515 01:55:25.073895 231762 net.cpp:265] Creating Layer pool5
      I0515 01:55:25.073899 231762 net.cpp:1264] pool5 <- conv5
      I0515 01:55:25.073902 231762 net.cpp:1238] pool5 -> pool5
      I0515 01:55:25.073909 231762 net.cpp:360] Setting up pool5
      I0515 01:55:25.073917 231762 net.cpp:367] Top shape: 16 256 9 9 (331776)
      I0515 01:55:25.073921 231762 net.cpp:375] Memory required for data: 142532672
      I0515 01:55:25.073923 231762 layer_factory.hpp:114] Creating layer ip6
      I0515 01:55:25.073932 231762 net.cpp:265] Creating Layer ip6
      I0515 01:55:25.073935 231762 net.cpp:1264] ip6 <- pool5
      I0515 01:55:25.073938 231762 net.cpp:1238] ip6 -> ip6
      I0515 01:55:25.126351 231762 net.cpp:360] Setting up ip6
      I0515 01:55:25.126366 231762 net.cpp:367] Top shape: 16 512 (8192)
      I0515 01:55:25.126370 231762 net.cpp:375] Memory required for data: 142565440
      I0515 01:55:25.126380 231762 layer_factory.hpp:114] Creating layer relu6
      I0515 01:55:25.126389 231762 net.cpp:265] Creating Layer relu6
      I0515 01:55:25.126391 231762 net.cpp:1264] relu6 <- ip6
      I0515 01:55:25.126396 231762 net.cpp:1225] relu6 -> ip6 (in-place)
      I0515 01:55:25.126404 231762 net.cpp:360] Setting up relu6
      I0515 01:55:25.126405 231762 net.cpp:367] Top shape: 16 512 (8192)
      I0515 01:55:25.126408 231762 net.cpp:375] Memory required for data: 142598208
      I0515 01:55:25.126412 231762 layer_factory.hpp:114] Creating layer drop6
      I0515 01:55:25.126420 231762 net.cpp:265] Creating Layer drop6
      I0515 01:55:25.126421 231762 net.cpp:1264] drop6 <- ip6
      I0515 01:55:25.126425 231762 net.cpp:1225] drop6 -> ip6 (in-place)
      I0515 01:55:25.126433 231762 net.cpp:360] Setting up drop6
      I0515 01:55:25.126435 231762 net.cpp:367] Top shape: 16 512 (8192)
      I0515 01:55:25.126438 231762 net.cpp:375] Memory required for data: 142630976
      I0515 01:55:25.126456 231762 layer_factory.hpp:114] Creating layer ip7
      I0515 01:55:25.126466 231762 net.cpp:265] Creating Layer ip7
      I0515 01:55:25.126467 231762 net.cpp:1264] ip7 <- ip6
      I0515 01:55:25.126471 231762 net.cpp:1238] ip7 -> ip7
      I0515 01:55:25.127684 231762 net.cpp:360] Setting up ip7
      I0515 01:55:25.127692 231762 net.cpp:367] Top shape: 16 512 (8192)
      I0515 01:55:25.127696 231762 net.cpp:375] Memory required for data: 142663744
      I0515 01:55:25.127701 231762 layer_factory.hpp:114] Creating layer relu7
      I0515 01:55:25.127707 231762 net.cpp:265] Creating Layer relu7
      I0515 01:55:25.127713 231762 net.cpp:1264] relu7 <- ip7
      I0515 01:55:25.127717 231762 net.cpp:1225] relu7 -> ip7 (in-place)
      I0515 01:55:25.127722 231762 net.cpp:360] Setting up relu7
      I0515 01:55:25.127724 231762 net.cpp:367] Top shape: 16 512 (8192)
      I0515 01:55:25.127727 231762 net.cpp:375] Memory required for data: 142696512
      I0515 01:55:25.127729 231762 layer_factory.hpp:114] Creating layer drop7
      I0515 01:55:25.127737 231762 net.cpp:265] Creating Layer drop7
      I0515 01:55:25.127738 231762 net.cpp:1264] drop7 <- ip7
      I0515 01:55:25.127743 231762 net.cpp:1225] drop7 -> ip7 (in-place)
      I0515 01:55:25.127748 231762 net.cpp:360] Setting up drop7
      I0515 01:55:25.127750 231762 net.cpp:367] Top shape: 16 512 (8192)
      I0515 01:55:25.127753 231762 net.cpp:375] Memory required for data: 142729280
      I0515 01:55:25.127756 231762 layer_factory.hpp:114] Creating layer ip1
      I0515 01:55:25.127763 231762 net.cpp:265] Creating Layer ip1
      I0515 01:55:25.127766 231762 net.cpp:1264] ip1 <- ip7
      I0515 01:55:25.127770 231762 net.cpp:1238] ip1 -> ip1
      I0515 01:55:25.128365 231762 net.cpp:360] Setting up ip1
      I0515 01:55:25.128368 231762 net.cpp:367] Top shape: 16 250 (4000)
      I0515 01:55:25.128371 231762 net.cpp:375] Memory required for data: 142745280
      I0515 01:55:25.128376 231762 layer_factory.hpp:114] Creating layer loss
      I0515 01:55:25.128382 231762 net.cpp:265] Creating Layer loss
      I0515 01:55:25.128384 231762 net.cpp:1264] loss <- ip1
      I0515 01:55:25.128387 231762 net.cpp:1264] loss <- labels
      I0515 01:55:25.128391 231762 net.cpp:1238] loss -> loss
      I0515 01:55:25.128403 231762 layer_factory.hpp:114] Creating layer loss
      I0515 01:55:25.128424 231762 net.cpp:360] Setting up loss
      I0515 01:55:25.128427 231762 net.cpp:367] Top shape: (1)
      I0515 01:55:25.128429 231762 net.cpp:370]     with loss weight 1
      I0515 01:55:25.128458 231762 net.cpp:375] Memory required for data: 142745284
      I0515 01:55:25.128460 231762 net.cpp:437] loss needs backward computation.
      I0515 01:55:25.128464 231762 net.cpp:437] ip1 needs backward computation.
      I0515 01:55:25.128473 231762 net.cpp:437] drop7 needs backward computation.
      I0515 01:55:25.128475 231762 net.cpp:437] relu7 needs backward computation.
      I0515 01:55:25.128477 231762 net.cpp:437] ip7 needs backward computation.
      I0515 01:55:25.128480 231762 net.cpp:437] drop6 needs backward computation.
      I0515 01:55:25.128482 231762 net.cpp:437] relu6 needs backward computation.
      I0515 01:55:25.128484 231762 net.cpp:437] ip6 needs backward computation.
      I0515 01:55:25.128487 231762 net.cpp:437] pool5 needs backward computation.
      I0515 01:55:25.128490 231762 net.cpp:437] relu5 needs backward computation.
      I0515 01:55:25.128492 231762 net.cpp:437] conv5 needs backward computation.
      I0515 01:55:25.128495 231762 net.cpp:437] relu4 needs backward computation.
      I0515 01:55:25.128497 231762 net.cpp:437] conv4 needs backward computation.
      I0515 01:55:25.128500 231762 net.cpp:437] relu3 needs backward computation.
      I0515 01:55:25.128502 231762 net.cpp:437] conv3 needs backward computation.
      I0515 01:55:25.128505 231762 net.cpp:437] pool2 needs backward computation.
      I0515 01:55:25.128509 231762 net.cpp:437] relu2 needs backward computation.
      I0515 01:55:25.128510 231762 net.cpp:437] conv2 needs backward computation.
      I0515 01:55:25.128513 231762 net.cpp:437] pool1 needs backward computation.
      I0515 01:55:25.128515 231762 net.cpp:437] relu1 needs backward computation.
      I0515 01:55:25.128518 231762 net.cpp:437] conv1 needs backward computation.
      I0515 01:55:25.128520 231762 net.cpp:439] data does not need backward computation.
      I0515 01:55:25.128523 231762 net.cpp:481] This network produces output loss
      I0515 01:55:25.128535 231762 net.cpp:521] Network initialization done.
      I0515 01:55:25.128628 231762 solver.cpp:121] Solver scaffolding done.
      I0515 01:55:25.128671 231762 caffe.cpp:342] Starting Optimization
      I0515 01:55:25.128675 231762 solver.cpp:397] Solving sketch_a_net
      I0515 01:55:25.128679 231762 solver.cpp:398] Learning Rate Policy: fixed
      I0515 01:55:25.319639 231762 net.cpp:1453]     [Forward] Layer data, top blob data data: 0.00864378
      I0515 01:55:25.319705 231762 net.cpp:1453]     [Forward] Layer data, top blob labels data: 0.6875
      I0515 01:55:25.441561 231762 net.cpp:1453]     [Forward] Layer conv1, top blob conv1 data: 0.0134911
      I0515 01:55:25.441624 231762 net.cpp:1465]     [Forward] Layer conv1, param blob 0 data: 0.0235335
      I0515 01:55:25.441629 231762 net.cpp:1465]     [Forward] Layer conv1, param blob 1 data: 0
      I0515 01:55:25.442117 231762 net.cpp:1453]     [Forward] Layer relu1, top blob conv1 data: 0.0134911
      I0515 01:55:25.453485 231762 net.cpp:1453]     [Forward] Layer pool1, top blob pool1 data: 0.0483181
      I0515 01:55:25.460430 231762 net.cpp:1453]     [Forward] Layer conv2, top blob conv2 data: 0.0353069
      I0515 01:55:25.460469 231762 net.cpp:1465]     [Forward] Layer conv2, param blob 0 data: 0.0216604
      I0515 01:55:25.460474 231762 net.cpp:1465]     [Forward] Layer conv2, param blob 1 data: 0
      I0515 01:55:25.460602 231762 net.cpp:1453]     [Forward] Layer relu2, top blob conv2 data: 0.0353069
      I0515 01:55:25.461946 231762 net.cpp:1453]     [Forward] Layer pool2, top blob pool2 data: 0.0763257
      I0515 01:55:25.464470 231762 net.cpp:1453]     [Forward] Layer conv3, top blob conv3 data: 0.0477126
      I0515 01:55:25.464498 231762 net.cpp:1465]     [Forward] Layer conv3, param blob 0 data: 0.025507
      I0515 01:55:25.464504 231762 net.cpp:1465]     [Forward] Layer conv3, param blob 1 data: 0
      I0515 01:55:25.464586 231762 net.cpp:1453]     [Forward] Layer relu3, top blob conv3 data: 0.0477126
      I0515 01:55:25.468986 231762 net.cpp:1453]     [Forward] Layer conv4, top blob conv4 data: 0.0352706
      I0515 01:55:25.469017 231762 net.cpp:1465]     [Forward] Layer conv4, param blob 0 data: 0.0180323
      I0515 01:55:25.469022 231762 net.cpp:1465]     [Forward] Layer conv4, param blob 1 data: 0
      I0515 01:55:25.469106 231762 net.cpp:1453]     [Forward] Layer relu4, top blob conv4 data: 0.0352706
      I0515 01:55:25.473371 231762 net.cpp:1453]     [Forward] Layer conv5, top blob conv5 data: 0.0241036
      I0515 01:55:25.473399 231762 net.cpp:1465]     [Forward] Layer conv5, param blob 0 data: 0.018016
      I0515 01:55:25.473414 231762 net.cpp:1465]     [Forward] Layer conv5, param blob 1 data: 0
      I0515 01:55:25.473495 231762 net.cpp:1453]     [Forward] Layer relu5, top blob conv5 data: 0.0241036
      I0515 01:55:25.474439 231762 net.cpp:1453]     [Forward] Layer pool5, top blob pool5 data: 0.0434049
      I0515 01:55:25.484984 231762 net.cpp:1453]     [Forward] Layer ip6, top blob ip6 data: 0.0582009
      I0515 01:55:25.485649 231762 net.cpp:1465]     [Forward] Layer ip6, param blob 0 data: 0.00601441
      I0515 01:55:25.485666 231762 net.cpp:1465]     [Forward] Layer ip6, param blob 1 data: 0
      I0515 01:55:25.485808 231762 net.cpp:1453]     [Forward] Layer relu6, top blob ip6 data: 0.0276806
      I0515 01:55:25.592556 231762 net.cpp:1453]     [Forward] Layer drop6, top blob ip6 data: 0.0280866
      I0515 01:55:25.592963 231762 net.cpp:1453]     [Forward] Layer ip7, top blob ip7 data: 0.0554552
      I0515 01:55:25.593001 231762 net.cpp:1465]     [Forward] Layer ip7, param blob 0 data: 0.0381965
      I0515 01:55:25.593013 231762 net.cpp:1465]     [Forward] Layer ip7, param blob 1 data: 0
      I0515 01:55:25.593166 231762 net.cpp:1453]     [Forward] Layer relu7, top blob ip7 data: 0.0269386
      I0515 01:55:25.593291 231762 net.cpp:1453]     [Forward] Layer drop7, top blob ip7 data: 0.0281175
      I0515 01:55:25.593519 231762 net.cpp:1453]     [Forward] Layer ip1, top blob ip1 data: 0.057124
      I0515 01:55:25.593544 231762 net.cpp:1465]     [Forward] Layer ip1, param blob 0 data: 0.0383157
      I0515 01:55:25.593605 231762 net.cpp:1465]     [Forward] Layer ip1, param blob 1 data: 0
      I0515 01:55:25.614830 231762 net.cpp:1453]     [Forward] Layer loss, top blob loss data: 5.50628
      I0515 01:55:25.614907 231762 net.cpp:1481]     [Backward] Layer loss, bottom blob ip1 diff: 0.000497967
      I0515 01:55:25.615401 231762 net.cpp:1481]     [Backward] Layer ip1, bottom blob ip7 diff: 0.00237027
      I0515 01:55:25.615443 231762 net.cpp:1492]     [Backward] Layer ip1, param blob 0 diff: 0.000222586
      I0515 01:55:25.615454 231762 net.cpp:1492]     [Backward] Layer ip1, param blob 1 diff: 0.00790243
      I0515 01:55:25.615480 231762 net.cpp:1481]     [Backward] Layer drop7, bottom blob ip7 diff: 0.00237809
      I0515 01:55:25.615677 231762 net.cpp:1481]     [Backward] Layer relu7, bottom blob ip7 diff: 0.00117207
      I0515 01:55:25.617923 231762 net.cpp:1481]     [Backward] Layer ip7, bottom blob ip6 diff: 0.00214139
      I0515 01:55:25.617995 231762 net.cpp:1492]     [Backward] Layer ip7, param blob 0 diff: 0.00041965
      I0515 01:55:25.618008 231762 net.cpp:1492]     [Backward] Layer ip7, param blob 1 diff: 0.0133229
      I0515 01:55:25.618032 231762 net.cpp:1481]     [Backward] Layer drop6, bottom blob ip6 diff: 0.00213438
      I0515 01:55:25.618234 231762 net.cpp:1481]     [Backward] Layer relu6, bottom blob ip6 diff: 0.00107521
      I0515 01:55:25.627077 231762 net.cpp:1481]     [Backward] Layer ip6, bottom blob pool5 diff: 0.000338665
      I0515 01:55:25.627769 231762 net.cpp:1492]     [Backward] Layer ip6, param blob 0 diff: 0.00043056
      I0515 01:55:25.627785 231762 net.cpp:1492]     [Backward] Layer ip6, param blob 1 diff: 0.00950258
      I0515 01:55:25.629072 231762 net.cpp:1481]     [Backward] Layer pool5, bottom blob conv5 diff: 7.43115e-05
      I0515 01:55:25.629312 231762 net.cpp:1481]     [Backward] Layer relu5, bottom blob conv5 diff: 4.90408e-05
      I0515 01:55:25.642890 231762 net.cpp:1481]     [Backward] Layer conv5, bottom blob conv4 diff: 0.000135426
      I0515 01:55:25.642947 231762 net.cpp:1492]     [Backward] Layer conv5, param blob 0 diff: 0.000684219
      I0515 01:55:25.642954 231762 net.cpp:1492]     [Backward] Layer conv5, param blob 1 diff: 0.0154824
      I0515 01:55:25.643112 231762 net.cpp:1481]     [Backward] Layer relu4, bottom blob conv4 diff: 6.55505e-05
      I0515 01:55:25.652350 231762 net.cpp:1481]     [Backward] Layer conv4, bottom blob conv3 diff: 9.23094e-05
      I0515 01:55:25.652405 231762 net.cpp:1492]     [Backward] Layer conv4, param blob 0 diff: 0.000582614
      I0515 01:55:25.652411 231762 net.cpp:1492]     [Backward] Layer conv4, param blob 1 diff: 0.00893011
      I0515 01:55:25.652551 231762 net.cpp:1481]     [Backward] Layer relu3, bottom blob conv3 diff: 4.3061e-05
      I0515 01:55:25.657308 231762 net.cpp:1481]     [Backward] Layer conv3, bottom blob pool2 diff: 8.62978e-05
      I0515 01:55:25.657349 231762 net.cpp:1492]     [Backward] Layer conv3, param blob 0 diff: 0.000614721
      I0515 01:55:25.657356 231762 net.cpp:1492]     [Backward] Layer conv3, param blob 1 diff: 0.00550787
      I0515 01:55:25.658246 231762 net.cpp:1481]     [Backward] Layer pool2, bottom blob conv2 diff: 1.87377e-05
      I0515 01:55:25.658514 231762 net.cpp:1481]     [Backward] Layer relu2, bottom blob conv2 diff: 1.17266e-05
      I0515 01:55:25.670866 231762 net.cpp:1481]     [Backward] Layer conv2, bottom blob pool1 diff: 3.91207e-05
      I0515 01:55:25.670902 231762 net.cpp:1492]     [Backward] Layer conv2, param blob 0 diff: 0.000793592
      I0515 01:55:25.670908 231762 net.cpp:1492]     [Backward] Layer conv2, param blob 1 diff: 0.00641661
      I0515 01:55:25.684468 231762 net.cpp:1481]     [Backward] Layer pool1, bottom blob conv1 diff: 8.41338e-06
      I0515 01:55:25.685132 231762 net.cpp:1481]     [Backward] Layer relu1, bottom blob conv1 diff: 1.27889e-06
      I0515 01:55:25.791841 231762 net.cpp:1492]     [Backward] Layer conv1, param blob 0 diff: 0.000298852
      I0515 01:55:25.791887 231762 net.cpp:1492]     [Backward] Layer conv1, param blob 1 diff: 0.00304507
      E0515 01:55:25.794600 231762 net.cpp:1603]     [Backward] All net params (data, diff): L1 norm = (114025, 5848.87); L2 norm = (47.2451, 4.0431)
      I0515 01:55:25.800848 231762 solver.cpp:312] Iteration 0, loss = 5.50628
      I0515 01:55:25.800858 231762 solver.cpp:333]     Train net output #0: loss = 5.50628 (* 1 = 5.50628 loss)
      I0515 01:55:25.800868 231762 sgd_solver.cpp:215] Iteration 0, lr = 0.01
      I0515 01:55:28.151196 231762 net.cpp:1453]     [Forward] Layer data, top blob data data: 0.00902651
      I0515 01:55:28.151252 231762 net.cpp:1453]     [Forward] Layer data, top blob labels data: 1.1875
      I0515 01:55:28.249269 231762 net.cpp:1453]     [Forward] Layer conv1, top blob conv1 data: 0.0192651
      I0515 01:55:28.249332 231762 net.cpp:1465]     [Forward] Layer conv1, param blob 0 data: 0.0235447
      I0515 01:55:28.249341 231762 net.cpp:1465]     [Forward] Layer conv1, param blob 1 data: 0.00601711
      I0515 01:55:28.249671 231762 net.cpp:1453]     [Forward] Layer relu1, top blob conv1 data: 0.0192651
      *** Aborted at 1526374528 (unix time) try "date -d @1526374528" if you are using GNU date ***
      PC: @     0x1549fe404ce4 caffe::MKLDNNMemoryDescriptorBase<>::prv_size()
      *** SIGSEGV (@0x0) received by PID 231762 (TID 0x154a00bf9780) from PID 0; stack trace: ***
          @     0x1549fa01e5e0 (unknown)
          @     0x1549fe404ce4 caffe::MKLDNNMemoryDescriptorBase<>::prv_size()
          @     0x1549fe404cd7 caffe::MKLDNNMemoryDescriptorBase<>::prv_count()
          @     0x1549feb98931 caffe::Blob<>::asum_data()
          @     0x1549fe42d145 caffe::Net<>::Forward()
          @     0x1549fe3d1f8d caffe::Solver<>::ForwardBackward()
          @     0x1549fe3c4822 boost::detail::function::function_obj_invoker0<>::invoke()
          @     0x1549fe3d1de7 boost::function0<>::operator()()
          @     0x1549fe3be724 _ZN5caffe6SolverIfE4StepEi.h
          @     0x1549fe3bd887 _ZN5caffe6SolverIfE5SolveEPKc.h
          @     0x55dc1d50203c train()
          @     0x55dc1d5004f2 main
          @     0x1549f9c6dc05 __libc_start_main
          @     0x55dc1d5002e9 (unknown)
      /var/spool/torque/mom_priv/jobs/82540.c009.SC: line 6: 231762 Segmentation fault      caffe train -solver ./sketch_solver_.prototxt