Rotation matrix of triangle in 3D

Question

How can I find out the rotation matrix of a right angle triangle defined by 3 points in 3D space (assuming the un-rotated triangle faces the x axis)

Answer 1

Suppose the triangle has vertices $v_1,v_2,v_3$. Construct the unit normal:

$$ n =\frac{(v_2-v_1)\times(v_3-v_1)}{|(v_2-v_1)\times(v_3-v_1)|} $$

We want to rotate $n$ to $x=(1,0,0)$. First we define the rotation axis:

$$ k=\frac{n\times x }{ |n\times x|} $$

and the rotation angle

$$\theta=\cos^{-1}(n\cdot x)$$

You can then either construct the 3D rotation matrix given here, or simply use the Rodrigues' rotation formula to rotate each of the vertices:

$$ v_i' = v_i\cos\theta + (k\times v_i)\sin\theta + k(k\cdot v_i)(1-\cos \theta) $$