The way one does this is to start from the symmetry group $\mathrm{SO}(3)$ of rotations in 3-dimensional space, consisting of the $3\times 3$ real, orthogonal matrices with determinant +1. The infinitesimal generators of this group constitute the Lie algebra $\mathfrak{so}(3)$, which is the set of $3\times 3$ antisymmetric matrices. The standard basis for this algebra is
$$\mathbb J_1=\pmatrix{0 & 0 & 0 \\ 0 & 0 &-1\\0&1&0} \quad \mathbb J_2=\pmatrix{0 & 0 & 1 \\ 0& 0 &0\\-1&0&0} \quad \mathbb J_3=\pmatrix{0 & -1 & 0 \\ 1 & 0 &0\\0&0&0}$$
with commutation relations $[\mathbb J_i,\mathbb J_j]=\epsilon_{ijk}\mathbb J_k$.
If you are unfamiliar with any of this, it suffices to say that if we have some Hilbert space $\mathscr H$ and we can find a set of three self-adjoint operators $\{\hat J_1,\hat J_2, \hat J_3\}$ with the same commutation relations (that is, $[\hat J_i,\hat J_j]=\epsilon_{ijk}\hat J_k$), then those operators can be understood as the angular momentum operators on $\mathscr H$.
The map $\rho: \mathbb J_i \mapsto \hat J_i$ which takes an abstract element of $\mathfrak{so}(3)$ to a concrete operator on some Hilbert space is called a representation; a classification of all such representations would tell us how to implement angular momentum operators on whatever Hilbert space we want.
The representation theory of the rotation group $\mathrm{SO}(3)$ and its Lie algebra $\mathfrak{so}(3)$ is a topic which could occupy a large section of a textbook in its own right, so a full and honest explanation is far beyond the scope of a single PSE answer. However, we can quote the salient information.
As a preliminary note, all finite-dimensional complex Hilbert spaces are isomorphic to $\mathbb C^n$ for some $n$, so those are the finite-dimensional Hilbert spaces we'll speak of. Secondly, you can always chain together two representations to get a larger one - for example, you could take the operators $J^{(2)}_i$ which act on $\mathbb C^2$ and $J^{(3)}_i$ which act on $\mathbb C^3$ and put them together in a block-diagonal way to produce operators which act on $\mathbb C^5$ like this:
$$ \pmatrix{J_i^{(2)} & \matrix{0&0&0\\0&0&0}\\\matrix{0&0\\0&0\\0&0} & J^{(3)}_i}$$
However, that doesn't really give us anything new, so we will ignore all such so-called reducible representations and focus on the irreducible ones.
With those preliminaries out of the way, the important results are as follows:
- For each $n>0$, there is (up to change-of-basis) exactly one way to represent $\mathfrak{so}(3)$ as self-adjoint operators on $\mathbb C^n$.
- For any finite-dimensional irreducible representation $\{\hat J_1,\hat J_2,\hat J_3\}$, the operator $\hat J^2 := \hat J_1^2 +\hat J_2^2 + \hat J_3^2$ commutes with each $\hat J_i$ and is in fact proportional to the identity operator, $\hat J^2 = c \mathbb I$. The constant $c$ depends on the representation, and can be shown to be equal to $j(j+1)$ for some non-negative integer or half-integer $j$; furthermore, the eigenvalues of each $j_i$ can be shown to range from $-j$ to $j$ in integer steps.
- The aforementioned $j$ turns out to be related to the dimensionality $n$ of the representation space via $n = 2j+1$. We use $j$ to label the particles in question, calling them "spin-$j$."
- The infinite dimensional case is a bit more subtle, but we can also represent $\mathfrak{so}(3)$ on infinite dimensional Hilbert spaces like $L^2(\mathbb R^3)$. Such representations can be obtained e.g. by implementing rotations as families of unitary operators and then finding the corresponding self-adjoint generators via Stone's theorem. However, note that $J^2$ is no longer proportional to the identity operator in the infinite-dimensional case.
Are there convincing arguments that suggest taking $\mathbb C^{2s+1}$ as Hilbert spaces for the spin angular momentum operator $\hat S$?
Experiments show us that certain particles are not accurately modeled by wavefunctions which inhabit the Hilbert space $L^2(\mathbb R^3)$. For example, an electron inhabiting the $\mathrm{1s}$ orbital of the hydrogen atom possesses non-zero angular momentum despite the fact that the angular momentum associated to its spatial wavefunction is equal to zero. Furthermore, there are only two distinct eigenvalues for this intrinsic angular momentum possessed by an electron, which leads us to surmise that the correct thing to do is to take the full Hilbert space to be $L^2(\mathbb R^3)\otimes \mathbb C^2$.
From there, we understand angular momentum as having an extrinsic part $\hat L_i$ associated to the wavefunction $\in L^2(\mathbb R^3)$ and an intrinsic part $\hat S_i$ associated to the internal spin state $\in \mathbb C^2$.