Relative orientation is to find the rotations and transitions of an image with respect to a reference photo. So, in the problem of relative orientation, we are always faced with a pair of images(Stereo pair).
Knowns in Relative orientation problem includes camera calibration parameters and a set of common image points that are read or automatically extracted between the pair. The unknowns include the position and the relative orientation of the second photo compared to the first photo. Therefore, in the problem of relative orientation, 5 degrees of freedom (relative angles of the second photo, and base of the model) could be considered.
Geometric constraints: In the problem of relative orientation, the first photo is considered as the origin of the “3D Cartesian coordinate system without distortion as discussed before” and the length of the base is assumed to be constant. However, other restrictions are easily defined and do not cause any issue in solving the problem.
If the initial values of the parameters are known to us with acceptable accuracy, we can use collinearity equations, otherwise we use coplanarity condition to solve this problem. The simple form of the collinearity equations is
where x is the coordinate of the photo point in the 3D coordinate system without distortion, R is the 3D rotation matrix of the photo in the model coordinate system, X is the coordinate of the model point in the model coordinate system, and X_0 is the shift of the photo relative to the origin of the model. We can rewrite the above relationship based on the image point in the form below
If we open the elements of this matrix equation, we will have
To remove the third element, we divide all three elements by it
In this equation, x’ is the image coordinate of the model point in the 3D coordinate system without distortion. With this division, we reached to a simple form for the equations
A point to consider here is that instead of dividing all the elements by z, we could divide them by -z. It can even be said that this division is even closer to the reality of our problem, because the ideal camera coordinate system in direct mode is easier to understand than the ideal camera coordinate system in reverse mode, although both coordinate systems are ultimately equivalent.
So, in direct mode we have
Now let’s see how the equations of the collinearity are used in the solution of relative orientation. If we consider the equations with all possible parameters, we get the following form
Parameters
As we can see, in equations 13 and 14, each equation is a non-linear function consisting of 20 parameters. In total, we have 21 parameters in the colinear condition equations, among which are two parameters of image observations, 10 parameters of internal orientation, 6 parameters of external orientation, and 3 parameters of model coordinates. In this formulating method, a simplification has also been made in such a way that the unknown coordinates of model points are considered as independent parameters, however, these parameters themselves are a function of other parameters. This simplification makes it much easier for us to use them inside equations.
Solutions
Solving the problem of relative orientation is analytically done by using the equations of collinearity or coplanarity. Relative orientation can be solved in several ways using the collinearity condition: 1- Right-sided relative orientation: in which the rotation and translation parameters of the left image are assumed to be fixed and only the unknowns of the problem will be related to the rotational and translational parameters of the right image. and 2- the left-sided relative orientation in which the rotational and translational parameters of the right image are assumed to be fixed and only the unknowns of the problem related to the rotational and translational parameters of the left image are considered. This type of naming is only used in aerial photogrammetry, basically one photo can be called the left photo and the other photo the right photo. In the short-range mode, the interpretation of left and right in the case of a pair of stereo images will be meaningless. Since 5 unknown parameters are considered, one unknown parameter can be reduced in two ways: 1- By assuming a fixed base length. 2- By assuming the largest component of the base to be constant, for example, if X_0 is greater than Y_0 and Z_0, then the value of X_0 is assumed to be constant. Implementing the second method is relatively easier compared to the first method.
In solving relative orientation with collinearity equations, a basic assumption is that the initial values of the unknowns are known with sufficient accuracy. If the unknowns are not known, solving the equations will fail. For this purpose, we use the Taylor expansion of each function to linearize the equations.
Solving relative justification with collinearity equations
To solve the relative orientation using the collinearity condition, we use equations 13 and 14. In this case, we first obtain the Taylor expansion of the functions, then we move towards an optimal point using the partial derivatives of a relatively correct estimate in the parameter space.
If we assume that we want to do a one-way relative orientation with the elements of the second image, provided that we want to consider the value of 〖X_0〗_2 constant, then our unknowns are
Now we can form A matrix
In the above equation, we have assumed that there are m model points and as a result we have 2m corresponding tie points.Therefore, the number of equations will be 4m. Here, the equations corresponding to the jth model point in the first photo are shown with 〖f_j1〗_x and 〖f_j1〗_y, and these equations for the jth model point in the second photo are shown with 〖f_j2〗_x and 〖f_j2〗_y .
Therefore, as it is known in these symbols, the first four equations are related to the first tie point and the last four equations are related to the last tie point.
The order of the columns is such that the unknowns of relative orientation are placed first and then the coordinates of the tie points are placed
A sparse matrix is a matrix in which most of its entries are equal to zero. In general, it is easier to work with sparse matrices if an effective data structure is used to store and perform algebraic operations on them. Carefully in the matrix structure of relative orientation coefficients, we notice that the following sparse structure exists
If instead of calculating A, we can find a way to calculate A^T A, we will reach a more optimal structure as shown below.
As we can see, the above structure is sparse, and the dimensions of this structure are much smaller, which will result in less needed storage and higher speed in least squares calculations.
Using each of the above two structures in a convex optimization process will help us to calculate the relative orientation elements.