Problems setting resolution and cropping for camera

[Astrocyte101]

Hello, I’m new to DeepLabCut-Live as well as to GitHub. Please kindly forgive me if I miss any guidelines. Thank you in advance for your patience and help.

Recently while trying to set up camera in the GUI, I found that if the resolution is set other than 640,480, the pop-up window would be entirely black when I click on the “Init Cam” button. However, the default settings (with a resolution of 640, 480) work well. Similar problems also arise when values are entered into “crop” (under the condition that resolution is set as 640, 480 as default).

I've posted this issue on image.sc (https://forum.image.sc/t/deeplabcut-live-gui-camera-cannot-change-resolution-or-crop/61488) about two weeks ago, and just found that GitHub also provides this Issues section. Sorry for not have noticed this. During these two weeks I have managed to solve some of the problems but encountered more. I'll describe them in detail below (those described in the post on image.sc are also included).

(For debugging purpose, I have created another environment and made some modifications to the codes. Please accept my apology if this is inappropriate.)

System information
Operating system: Windows 10 64-bit
Python version: 3.7.11
OpenCV version: originally 4.5.4.60 but changed to 4.5.5.62 (Please see below for details)
Camera: USB cameras purchased from local manufacturer (it seems that they do not have a brand name in English), the cameras can be controlled by OpenCV

(1) Problem setting resolution
I’ve first tested possible resolutions for my camera as in https://www.learnpythonwithrune.org/find-all-possible-webcam-resolutions-with-opencv-in-python/. 640 * 480, 800 * 600, 1024 * 768, 1280 *720, 1280 * 960, and 1920 * 1080 are all possible resolutions. (This test was run in the dlc-live environment)

However, if the resolution is set as 800, 600 in the GUI, there would be error messages:

Traceback (most recent call last):
  File "C:\Users\zoosh\anaconda3\envs\dlc-live\lib\site-packages\multiprocess\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\zoosh\anaconda3\envs\dlc-live\lib\site-packages\multiprocess\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\zoosh\anaconda3\envs\dlc-live\lib\site-packages\dlclivegui\camera_process.py", line 99, in _run_capture
    self._capture_loop()
  File "C:\Users\zoosh\anaconda3\envs\dlc-live\lib\site-packages\dlclivegui\camera_process.py", line 120, in _capture_loop
    np.copyto(self.frame, frame)
  File "<__array_function__ internals>", line 6, in copyto
ValueError: could not broadcast input array from shape (480,640,3) into shape (600,800,3)

If the resolution is set as other values, similar errors would also arise like:
ValueError: could not broadcast input array from shape (480,640,3) into shape (height,width,3)

I created another environment for debugging and added some codes to check if self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.im_size[0]) and self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.im_size[1]) actually works. It turns out that although self.im_size is successfully set as 800, 600, self.cap.get(cv2.CAP_PROP_FRAME_WIDTH) and self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT) still returned 640, 480. However, when I go into python -i the self.cap.set() commands are able to work, so I'm not sure about what's going wrong.

After that I upgraded the opencv-python from version 4.5.4.60 to version 4.5.5.62, and the CAP_PROP_FRAME_WIDTH and CAP_PROP_FRAME_HEIGHT can be successfully set as values other that 640,480 now. However, another problem showed up:

Traceback (most recent call last):
  File "C:\Users\zoosh\anaconda3\envs\dlc-live-debug\lib\site-packages\multiprocess\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\zoosh\anaconda3\envs\dlc-live-debug\lib\site-packages\multiprocess\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\zoosh\anaconda3\envs\dlc-live-debug\lib\site-packages\dlclivegui\camera_process.py", line 99, in _run_capture
    self._capture_loop()
  File "C:\Users\zoosh\anaconda3\envs\dlc-live-debug\lib\site-packages\dlclivegui\camera_process.py", line 116, in _capture_loop
    frame, frame_time = self.device.get_image_on_time()
  File "C:\Users\zoosh\anaconda3\envs\dlc-live-debug\lib\site-packages\dlclivegui\camera\opencv.py", line 154, in get_image_on_time
    ret, frame = self.cap.read()
cv2.error: OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\core\src\matrix.cpp:438: error: (-215:Assertion failed) _step >= minstep in function 'cv::Mat::Mat'

This error showed up for every acceptable resolution value except for 640, 480. However, when I switched to another USB camera that allowed a resolution of 320, 240, this error would not show up, but what the pop-up window showed was the upper-left part of the 640*480 frame.

It seemed to me that there might be something wrong related to OpenCV. However, I have no idea how to fix it. It would really be of great help if there's a solution to it. Many thanks in advance.

(2) Problem cropping
Under the condition that resolution is set as 640, 480, when crop is set as 0, 320, 0, 240, error messages showed:
ValueError: could not broadcast input array from shape (240,320,3) into shape (320,240,3)
After inspecting into the codes, I found that in the function set_im_size of Camera object, the variable self.im_size would be set as [240,320] if 0,320,0,240 were passed to ‘‘crop’’ in the GUI. This seems to eventually cause the dimensions of self.frame to be set as (320,240,3), which would be incompatible with the dimensions of variable frame as it should be (240,320,3).

In the environment for debugging, I tried to modify the below codes:

self.im_size = (
    (int(res[0]), int(res[1]))
    if self.crop is None
    else (self.crop[3] - self.crop[2], self.crop[1] - self.crop[0])
)

into:

self.im_size = (
    (int(res[0]), int(res[1]))
    if self.crop is None
    else (self.crop[1] - self.crop[0], self.crop[3] - self.crop[2])
)

and the cropping finally works.

However, when the opencv-python was upgraded to 4.5.5.62, another issue showed up that self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.im_size[0]) and self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.im_size[1]) would try to set the frame size as the size after cropping, which would actually lead to a nearest possible resolution being set.

(This would not happen in opencv-python 4.5.4.60 since the camera resolution cannot be set other than 640*480.)

For example, if I use the camera that also allows 320*240, when setting resolution as 640, 480 and crop as 210,530,90,410, the error message would be:
ValueError: could not broadcast input array from shape (150,110,3) into shape (320,320,3)
And self.cap.get(cv2.CAP_PROP_FRAME_WIDTH) and self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT) gave a resolution of 320, 240.

I tried to resolve this by modifying the values passed to self.im_size again:

self.im_size = (
    (int(res[0]), int(res[1]), int(res[0]), int(res[1]))
    if self.crop is None
    else (self.crop[1] - self.crop[0], self.crop[3] - self.crop[2], int(res[0]), int(res[1]))
)

and went to set_capture_device under opencv.py to set:

if self.im_size:
    self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.im_size[2])
    self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.im_size[3])

This solved the problem. self.cap.get(cv2.CAP_PROP_FRAME_WIDTH) and self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT) now give a resolution of 640, 480 and the frame can be cropped and displayed adequately.

Summary
The resolution problem still cannot be solved, probably due to some problems related to OpenCV. The cropping problem seemed to be able to be solved via above-mentioned modifications to the codes. As for now I have no idea what I can do for the resolution problem. I would really be grateful if these problems can be fixed. Also, I'm not sure whether my modifications for solving the cropping problem can be compatible with other functions in the packages. If you would like to consider fixing these problems in the official codes, it would really be of great help. Many many thanks and best regards.

[nithikc]

I am also having these same issues.