Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multi_stage_meanfield.cpp:49] This implementation has not been tested batch size > 1. #136

Open
fangxu622 opened this issue Jul 5, 2017 · 9 comments
Open

multi_stage_meanfield.cpp:49] This implementation has not been tested batch size > 1. #136

fangxu622 opened this issue Jul 5, 2017 · 9 comments

Comments

@fangxu622
Copy link

fangxu622 commented Jul 5, 2017 •
edited
Loading

Dear @bittnt ,

During training my CRFasRNN model, when I entered the command：
python ../fcn.berkeleyvision.org/voc-fcn8s-atonce/solve.py

This problem has occurred

I0705 20:45:21.787677 25165 net.cpp:111] Creating Layer inference1
I0705 20:45:21.787684 25165 net.cpp:478] inference1 <- unary
I0705 20:45:21.787703 25165 net.cpp:478] inference1 <- Q0
I0705 20:45:21.787719 25165 net.cpp:478] inference1 <- data_data_0_split_2
I0705 20:45:21.787731 25165 net.cpp:434] inference1 -> pred
I0705 20:45:21.787744 25165 multi_stage_meanfield.cpp:49] This implementation has not been tested batch size > 1.

this is my train.prototxt, it's TVG_CRFRNN_new_traintest.prototxt that replace data layer with fcn8s-atonce's data layer

layer {
  name: "data"
  type: "Python"
  top: "data"
  top: "label"
  python_param {
    module: "voc_layers"
    layer: "SBDDSegDataLayer"
    param_str: "{\'sbdd_dir\': \'../fcn.berkeleyvision.org/data/fcn-zy3-512-1024/traindata\', \'seed\': 1337, \'split\': \'train\', \'mean\': (109.08637, 105.57667, 89.27919)}"
  }
}


layer {
  name: "conv1_1"
  type: "Convolution"
  bottom: "data"
  top: "conv1_1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 64
    pad: 100
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu1_1"
  type: "ReLU"
  bottom: "conv1_1"
  top: "conv1_1"
}
layer {
  name: "conv1_2"
  type: "Convolution"
  bottom: "conv1_1"
  top: "conv1_2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 64
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu1_2"
  type: "ReLU"
  bottom: "conv1_2"
  top: "conv1_2"
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1_2"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "conv2_1"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2_1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 128
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu2_1"
  type: "ReLU"
  bottom: "conv2_1"
  top: "conv2_1"
}
layer {
  name: "conv2_2"
  type: "Convolution"
  bottom: "conv2_1"
  top: "conv2_2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 128
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu2_2"
  type: "ReLU"
  bottom: "conv2_2"
  top: "conv2_2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2_2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "conv3_1"
  type: "Convolution"
  bottom: "pool2"
  top: "conv3_1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu3_1"
  type: "ReLU"
  bottom: "conv3_1"
  top: "conv3_1"
}
layer {
  name: "conv3_2"
  type: "Convolution"
  bottom: "conv3_1"
  top: "conv3_2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu3_2"
  type: "ReLU"
  bottom: "conv3_2"
  top: "conv3_2"
}
layer {
  name: "conv3_3"
  type: "Convolution"
  bottom: "conv3_2"
  top: "conv3_3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu3_3"
  type: "ReLU"
  bottom: "conv3_3"
  top: "conv3_3"
}
layer {
  name: "pool3"
  type: "Pooling"
  bottom: "conv3_3"
  top: "pool3"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "conv4_1"
  type: "Convolution"
  bottom: "pool3"
  top: "conv4_1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu4_1"
  type: "ReLU"
  bottom: "conv4_1"
  top: "conv4_1"
}
layer {
  name: "conv4_2"
  type: "Convolution"
  bottom: "conv4_1"
  top: "conv4_2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu4_2"
  type: "ReLU"
  bottom: "conv4_2"
  top: "conv4_2"
}
layer {
  name: "conv4_3"
  type: "Convolution"
  bottom: "conv4_2"
  top: "conv4_3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu4_3"
  type: "ReLU"
  bottom: "conv4_3"
  top: "conv4_3"
}
layer {
  name: "pool4"
  type: "Pooling"
  bottom: "conv4_3"
  top: "pool4"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "conv5_1"
  type: "Convolution"
  bottom: "pool4"
  top: "conv5_1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu5_1"
  type: "ReLU"
  bottom: "conv5_1"
  top: "conv5_1"
}
layer {
  name: "conv5_2"
  type: "Convolution"
  bottom: "conv5_1"
  top: "conv5_2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu5_2"
  type: "ReLU"
  bottom: "conv5_2"
  top: "conv5_2"
}
layer {
  name: "conv5_3"
  type: "Convolution"
  bottom: "conv5_2"
  top: "conv5_3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    engine: CAFFE
  }
}
layer {
  name: "relu5_3"
  type: "ReLU"
  bottom: "conv5_3"
  top: "conv5_3"
}
layer {
  name: "pool5"
  type: "Pooling"
  bottom: "conv5_3"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "fc6"
  type: "Convolution"
  bottom: "pool5"
  top: "fc6"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 4096
    kernel_size: 7
    engine: CAFFE
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "drop6"
  type: "Dropout"
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc7"
  type: "Convolution"
  bottom: "fc6"
  top: "fc7"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 4096
    kernel_size: 1
    engine: CAFFE
  }
}
layer {
  name: "relu7"
  type: "ReLU"
  bottom: "fc7"
  top: "fc7"
}
layer {
  name: "drop7"
  type: "Dropout"
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "score-fr"
  type: "Convolution"
  bottom: "fc7"
  top: "score"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 11
    kernel_size: 1
    engine: CAFFE
    weight_filler {
      type: "msra"
      std: 0.010
    }
  }
}
layer {
  name: "score2"
  type: "Deconvolution"
  bottom: "score"
  top: "score2"
  param {
    lr_mult: 1
  }
  convolution_param {
    num_output: 11
    kernel_size: 4
    stride: 2
    weight_filler: { type: "bilinear" }
  }
}
layer {
  name: "score-pool4"
  type: "Convolution"
  bottom: "pool4"
  top: "score-pool4"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 11
    kernel_size: 1
    engine: CAFFE
    weight_filler {
      type: "msra"
      std: 0.010
    }
  }
}
layer { type: 'Crop' name: 'crop' bottom: 'score-pool4' bottom: 'score2'
  top: 'score-pool4c' }
layer {
  name: "fuse"
  type: "Eltwise"
  bottom: "score2"
  bottom: "score-pool4c"
  top: "score-fused"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "score4"
  type: "Deconvolution"
  bottom: "score-fused"
  top: "score4"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  convolution_param {
    num_output: 11
    bias_term: false
    kernel_size: 4
    stride: 2
    weight_filler: { type: "bilinear" }
  }
}
layer {
  name: "score-pool3"
  type: "Convolution"
  bottom: "pool3"
  top: "score-pool3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 11
    kernel_size: 1
    engine: CAFFE
    weight_filler {
      type: "msra"
      std: 0.010
    }
  }
}
layer { type: 'Crop' name: 'crop' bottom: 'score-pool3' bottom: 'score4'
  top: 'score-pool3c' }
layer {
  name: "fuse"
  type: "Eltwise"
  bottom: "score4"
  bottom: "score-pool3c"
  top: "score-final"
  eltwise_param {
    operation: SUM
  }
}
layer {
  name: "upsample"
  type: "Deconvolution"
  bottom: "score-final"
  top: "bigscore"
  param {
    lr_mult: 0
  }
  convolution_param {
    num_output: 11
    bias_term: false
    kernel_size: 16
    stride: 8
    weight_filler: { type: "bilinear" }
  }
}
layer { type: 'Crop' name: 'crop' bottom: 'bigscore' bottom: 'data' top: 'coarse' }

layer { type: 'Split' name: 'splitting'
  bottom: 'coarse' top: 'unary' top: 'Q0'
}

layer {
  name: "inference1"#if you set name "inference1", code will load parameters from caffemodel.
  type: "MultiStageMeanfield"
  bottom: "unary"
  bottom: "Q0"
  bottom: "data"
  top: "pred"
  param {
    lr_mult: 10000#learning rate for W_G
  }
  param {
  lr_mult: 10000#learning rate for W_B
  }
  param {
  lr_mult: 1000 #learning rate for compatiblity transform matrix
  }
  multi_stage_meanfield_param {
   num_iterations: 10
   compatibility_mode: POTTS#Initialize the compatilibity transform matrix with a matrix whose diagonal is -1.
   threshold: 2
   theta_alpha: 160
   theta_beta: 3
   theta_gamma: 3
   spatial_filter_weight: 3
   bilateral_filter_weight: 5
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "pred"
  bottom: "label"
  top: "loss"
  loss_param {
    ignore_label: 255
    normalize: false
  }
}

my solver.prototxt

train_net: "../fcn.berkeleyvision.org/voc-fcn8s-atonce/train.prototxt"
test_net: "../fcn.berkeleyvision.org/voc-fcn8s-atonce/val.prototxt"
test_iter: 847
# make test net, but don't invoke it from the solver itself
test_interval: 999999999
display: 20
average_loss: 20
lr_policy: "fixed"
# lr for unnormalized softmax
base_lr: 1e-10
# high momentum
momentum: 0.99
# no gradient accumulation
iter_size: 1
max_iter: 20000
weight_decay: 0.0005
snapshot: 5000
snapshot_prefix: "../fcn.berkeleyvision.org/voc-fcn8s-atonce/snapshot-crfasrnn/train"
test_initialization: false

this my solve.py

import sys

sys.path.append('./python')
sys.path.append('./python/caffe')

sys.path.append('/home/sensetime/DeepLearning/test/fcn.berkeleyvision.org')


import caffe
import surgery, score

import numpy as np
import os

try:
    import setproctitle
    setproctitle.setproctitle(os.path.basename(os.getcwd()))
except:
    pass

weights = '/home/sensetime/DeepLearning/CRFasRNN/crfasrnn/fcn.berkeleyvision.org/voc-fcn8s-atonce/snapshot/train_iter_15000.caffemodel'

# init
caffe.set_device(2)
caffe.set_mode_gpu()

solver = caffe.SGDSolver('../fcn.berkeleyvision.org/voc-fcn8s-atonce/solver.prototxt')
solver.net.copy_from(weights)

# surgeries
interp_layers = [k for k in solver.net.params.keys() if 'up' in k]
surgery.interp(solver.net, interp_layers)

# scoring
val = np.loadtxt('../fcn.berkeleyvision.org/data/fcn-zy3-512-1024/valdata/val.txt', dtype=str)

for _ in range(20):
    solver.step(1000)
    score.seg_tests(solver, False, val, layer='score')
@bittnt
Copy link
Collaborator

bittnt commented Jul 6, 2017

For this, you could get rid of the error message by doing either 1. remove that error message in the multi_stagemeanfield code 2) use batch_size = 1.

@fangxu622
Copy link
Author

The most critical thing is my batch_size=1., I use voc-layer load data。。。

@fangxu622
Copy link
Author

Now, I use the <Latest Caffe with CPU/GPU CRF-RNN> ，The problem has gone away and a new problem has come..

I0706 15:28:25.721647 31458 net.cpp:744] Ignoring source layer score_pool3
I0706 15:28:25.721652 31458 net.cpp:744] Ignoring source layer score_pool3c
I0706 15:28:25.721657 31458 net.cpp:744] Ignoring source layer upscore8
I0706 15:28:25.721662 31458 net.cpp:744] Ignoring source layer score
I0706 15:28:30.210567 31458 solver.cpp:218] Iteration 0 (-1.55027e-33 iter/s, 4.46282s/20 iters), loss = 2.51438e+06
I0706 15:28:30.210670 31458 solver.cpp:237]     Train net output #0: loss = 2.51438e+06 (* 1 = 2.51438e+06 loss)
I0706 15:28:30.210688 31458 sgd_solver.cpp:105] Iteration 0, lr = 1e-10
F0706 15:28:34.982275 31458 eltwise_layer.cpp:34] Check failed: bottom[0]->shape() == bottom[i]->shape() bottom[0]: 1 11 512 512 (2883584), bottom[1]: 1 11 1024 1024 (11534336)
*** Check failure stack trace: ***
Aborted (core dumped)

@fangxu622
Copy link
Author

the fcn-model is the voc-fcn8s-atonce,
I added the crfasRNN layer to fcn8s-atonce's train.prototxt , or the use TVG_CRFRNN_new_traintest.prototxt .it have the same problem...

@fangxu622
Copy link
Author

不晓得能和你讲中文吗？

@fangxu622
Copy link
Author

@bittnt

@fangxu622
Copy link
Author

I think the net bottom.shape is the same .but i don't know what something wrong

@fangxu622
Copy link
Author

fangxu622 commented Jul 6, 2017 •
edited
Loading

thanks , I have resolve it with resize the input image to (512,512).....

@matthiasmace matthiasmace mentioned this issue Nov 23, 2017
Open
@matthiasmace
Copy link

@bittnt
What do you mean ? How to remove je error message in the code please ? I tried but it returned compilation errors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bittnt @fangxu622 @matthiasmace