Let us look for a solution of the form:
$$\tag{1}\begin{pmatrix}x'\\ y'\end{pmatrix}=\begin{pmatrix}a & \ \ b \\b & -a\end{pmatrix}\begin{pmatrix}x\\ y\end{pmatrix}+\begin{pmatrix}e\\ f\end{pmatrix}$$
(see Edit 2 below)
(a symmetry matrix is orthogonal with determinant -1)
It suffices to write that points of intersection of the line with axes, i.e.,
$$\begin{pmatrix}x\\ y\end{pmatrix}=\begin{pmatrix}0\\ -4\end{pmatrix} \ \ \text{and} \ \ \begin{pmatrix}x\\ y\end{pmatrix}=\begin{pmatrix}-\tfrac{4}{3}\\ \ \ 0\end{pmatrix}$$
are invariant through this transformation, which gives, by plugging these expressions into (1), four equations with the 4 unknowns $a,b,c,d$:
$$\begin{cases}-4b+c&=&0\\ \ \ 4a+d&=&-4\\ \ \ \tfrac{4}{3}a+c&=&\tfrac{4}{3}\\ \ \ \tfrac{4}{3}b+d&=&0\end{cases}$$
out of which we obtain:
$$a=-4/5, b=3/5, c=12/5, d=-4/5.$$
Edit 1: Another solution (closer to what you have done).
We could have considered the succession of operations
$$\tag{3}S = T^{-1} \circ R^{-1} \circ S' \circ R \circ T \ \ \ \text{where}$$
T is the translation with vector $\binom{4}{0}$ bringing line with equation $y=3x-4$ onto line (D) with equation $y=3x$.
R is the rotation with angle -atan$(3)$ mapping (D) onto $x$ axis.
(we will denote by c and s resp. the cosine and sine of atan$(3)$):
S' is the (very simple) symmetry with respect to the $x$ axis.
Let us now give (3) under a matricial form:
$$\binom{x'}{y'}=\binom{0}{-4}+\underbrace{\begin{pmatrix}c&-s\\s&c\end{pmatrix}\begin{pmatrix}1& \ \ 0\\0&-1\end{pmatrix}\begin{pmatrix}c&s\\-s&c\end{pmatrix}}_{\begin{pmatrix}(c^2-s^2)&2cs\\2cs&-(c^2-s^2)\end{pmatrix}}(\binom{x}{y}+\binom{0}{4})$$
(the form we find here has exactly the same structure as the solution of Emilio Novati ; recall formulas $\cos^2a-\sin^2a=\cos(2a)$ and $2 \cos a \sin a = \sin(2a)$).
It is essential to understand that, in (3), you have to "follow the same way back" in the reverse order, of course.
Edit 2: There are 2 kinds of plane (vectorial) isometries : rotations, and symmetries with resp. expressions:
$$\tag{2}\begin{pmatrix}a & \ \ b \\-b & \ \ a\end{pmatrix} \ \ \ \text{and} \ \ \ \ \begin{pmatrix}a & \ \ b \\ \ \ b & -a\end{pmatrix} \ \ \ \text{with} \ \ \ a^2+b^2=1.$$
Why that ? Because an isometry has an orthogonal matrix, and an orthogonal matrix must be such that its column constitute an orthonormal basis.
Clearly, the first column is arbitrary. Let us denote by $\binom{a'}{b'}$ the second column of the matrix of an isometry. It must be orthogonal to the first one, i.e., dot product $\binom{a'}{b'} . \binom{a}{b}=0 \ \iff aa'+bb'=0$ with $a'^2+b'^2=1$. There are two solutions: $\binom{a'}{b'} = \binom{-b}{a}$ (direct orientation basis) or $\binom{a'}{b'} = \binom{b}{-a}$ (indirect orientation), whence formulas (2).
