There is a way to define the notion of prime number in the framework of group theory, thanks to the following observation:
Observation: A non-trivial group $G$ is cyclic of prime order iff for any subgroup $K \le G$ then $K=1$ or $G$.
Now the notion of group and subgroup can be internalized to any category (see group object and subobject). Then, using the above observation, we can internalize the notion of prime number to any category, and call it prime number object.
More precisely, let $C$ be a category with binary products and a terminal object $*$. Let $\operatorname{Grp}(C)$ be the category of group objects in $C$. An object $G$ in $\operatorname{Grp}(C)$ is called a prime number object in $C$ if for any monomorphism in $\operatorname{Grp}(C)$, $i: H \hookrightarrow G$, then $H=*$ or $i$ is an isomorphism.
Question: Is a prime number object always given by the data of a prime number (or a set of prime numbers)?
Remark 1: My guess is yes, but if we relax the notion of group object with the notion of Hopf object (as suggested in Adrien's answer to Is a Hopf algebra a group object of some category?) then I expect the question to become open and hard (see Is there a non-trivial Hopf algebra without left coideal subalgebra?).
Remark 2: We can also define the notion of natural number object (eventually relaxed as above) using the fact that a group is finite iff its subgroup lattice is finite, and Ore's theorem stating that a finite group is cyclic iff its subgroup lattice is distributive.