IR and RGB camera mapping

There are three coordinates system that relevant when we perform calibration, which are real world coordinates, camera matrices both IR camera and RGB camera. 

Suppose P is a point in real world coordinates, P_ir (3D point) is the same point but in the perspective of IR camera coordinates,  p_ir (2D point) is the same point in human perspective (the image). The relationship of P_ir and p_ir shows below:

$P_{ir} = {H_{ir}}^-1 p_{ir}$

$p_{ir} = {H_{ir}} P_{ir}$

Where H_ir is the intrinsic matrix of IR camera. which we obtain from previous calibration result.

P_ir can be transformed to RGB camera coordinates through the relative transformation R and T:

$P_{rgb} = R * P_{ir} + T$

Where R is the rotation matrix and T is the transition matrix.

Then we project P_rgb using H_rgb to obtain coordinates of P in RGB camera coordinates p_rgb.

$p_{rgb} = H_{rgb}P_{rgb}$

When two or more points projected on a same pixel, the closet point is chosen. 

It should be noticed that both p_rgb and p_ir are homogenous coordinates, therefore when we trying to form p_ir, we need to multiply the pixel coordinate (x,y) by z, which is the depth value. 

Find Rotation and Transition matrix

Extrinsic matrix transforms a point from 3D world coordinates to 3D camera coordinates. 

The extrinsic matrices we are going to use are different in format from the ones in previous post. The extrinsic matrices are 4*4 in size, which consist a 3*3 rotation matrix (R) , a 3*1 transition matrix (T) and a row of redundancies. The extrinsic matrices are showing as following:

IR camera:

-0.026084  -0.928295  -0.370929  194.092830

-0.999637  0.021708  0.015968  51.210837

-0.006771  0.371211  -0.928524  278.779239

0.000000  0.000000  0.000000  1.000000

RGB camera:

-0.036546  -0.929878  -0.366048  185.288116

-0.998892  0.023121  0.040994  28.217325

-0.029656  0.367140  -0.929693  268.344275

0.000000  0.000000  0.000000  1.000000

Suppose a 3D world coordinate P, it can be transformed to a 3D camera matrix using the matrices above. The following relations can be found:

$P_{ir} = R_{ir}P+T_{ir}$

$P_{rgb} = R_{rgb}P+T_{ir}$

We substitute P in equation 2 with P_ir,R_ir and T_ir from equation 1, we obtain:

$R = R_{rgb}R_{ir}^-1$

$T = T_{rgb} - R_{rgb}R_{ir}^-1T_{ir} = T_{rgb} - RT_{ir}$

Applying data, the results are:

R:

[0.99993188  0.01050178 -0.00504891]

[-0.01061398  0.99968556 -0.02271776]

[ 0.00480963  0.02276964  0.9997292 ]

T:

[-7.92176365]

[-14.58407103]

[-12.45903772]

The result looks well. However, GML tool manual suggested that calibration with two patterns at same time will produce a much more accurate result. Because we are going perform Feature Extraction in pixel level, a more accurate calibration with two different patterns may be required.