Base plane equation:

1. Collect a bunch of 3D points d_i in depth image space from the camera’s depth image.

2. Convert depth image-space points to metric camera space using camera’s intrinsic depth un-projection matrix (read from camera firmware): c_i = DP * d_i.

3. Find the plane equation (nx, ny, nz) * c_i = o that best fits all points c_i using a standard least-squares linear system solver A^T * A * x = A^T * b.

Projector calibration matrix:

1. Collect a set of tie points that associate a 2D point p_i in projector image space with a 3D point c_i in metric camera space.

2. Create an over-determined linear system for (p_ix * w_i, p_iy * w_i) = H_2 * c_i where H = (h0, …, h3, h4, …, h7, h8, …, h11) is an unknown 3×4 matrix, H_2 is the 2×4 matrix consisting of the upper two rows of H, and w_i = h8*c_ix + h9*c_iy + h10*c_iz + h11 is the homogeneous weight of H * (c_ix, c_iy, c_iz, 1).

3. Solve the system A^T * A * x = 0 by finding the eigenvector of A^T * A that has the smallest eigenvalue.

4. Extend the resulting matrix H to an OpenGL projection matrix by inserting a third row (0, 0, 0, -1) and pre-multiplying it with the inverse of a viewport matrix that maps the projector’s screen rectangle to [-1, 1] in both x and y, and the camera-space tie points’ z range, after multiplication with the extended H, to [-0.5, 0.5] in z. Z maps to [-0.5, 0.5] instead of [-1, 1] to avoid clipping away geometry that is lower or higher than the lowest or highest tie point, respectively.