unexpected patch extract phenomenon

[IQ17]

Hi, I notice an strange phenomenon in generating random affine transformed training patch pairs here

The synthesis affine transformation are composition of two matrices, being rot_LAFs and TA.
As TA is rectified to be UpIsUp, it will keep the vertical lines in images are vertical. And, the same rot_LAF is applied to anchor image and patch image, so both images will be rotated with the same angle.
Therefore, after applying rot_LAFs and TA, vertical lines in anchor image and positive image should be orientated to the same angle.

But the phenomenon is that, when apply rot_LAF and TA sequentially, this is true; while apply them jointly like in your code) , this is wrong.

code for appling rot_LAF and TA sequentially
def extract_random_LAF(data, max_rot = math.pi, max_tilt = 1.0, crop_size = 32): st = int((data.size(2) - crop_size)/2) fin = st + crop_size if type(max_rot) is float: rot_LAFs, inv_rotmat = get_random_rotation_LAFs(data, max_rot) else: rot_LAFs = max_rot inv_rotmat = None aff_LAFs, inv_TA = get_random_norm_affine_LAFs(data, max_tilt); # aff_LAFs[:,0:2,0:2] = torch.bmm(rot_LAFs[:,0:2,0:2],aff_LAFs[:,0:2,0:2]) # pdb.set_trace() data_aff = extract_patches(data, aff_LAFs, PS = data.size(2)) data_aff = extract_patches(data_aff, rot_LAFs, PS = data.size(2)) data_affcrop = data_aff[:,:, st:fin, st:fin].contiguous() return data_affcrop, data_aff, rot_LAFs,inv_rotmat,inv_TA

below are some examples, note that they obtained with different runs, so random angle and tilts are different

applying TA only (note TA is rectified to be UpIsUp)

applying rot_LAF only (note the same rot_LAFs is applied to both anchor and positive images)

applying rot_LAF and TA sequentially

applying rot_LAF and TA jointly

[rsokl]

An attempt to clean up OP's code-snippet.

def extract_random_LAF(data, max_rot = math.pi, max_tilt = 1.0, crop_size = 32): 
    st = int((data.size(2) - crop_size)/2) 
    fin = st + crop_size 
    if type(max_rot) is float: 
        rot_LAFs, inv_rotmat = get_random_rotation_LAFs(data, max_rot) 
    else: 
        rot_LAFs = max_rot 
        inv_rotmat = None 
    aff_LAFs, inv_TA = get_random_norm_affine_LAFs(data, max_tilt); 
    aff_LAFs[:,0:2,0:2] = torch.bmm(rot_LAFs[:,0:2,0:2],aff_LAFs[:,0:2,0:2])  # pdb.set_trace() 
    data_aff = extract_patches(data, aff_LAFs, PS = data.size(2)) 
    data_aff = extract_patches(data_aff, rot_LAFs, PS = data.size(2))   
    data_affcrop = data_aff[:,:, st:fin, st:fin].contiguous() 
    return data_affcrop, data_aff, rot_LAFs,inv_rotmat,inv_TA 

@ducha-aiki have you looked into this issue?