There is no mathematical definition of genetic signature. Please first have a look at Is there a formal definition of signature of natural selection?. As explain the term genetic signature is used in a very broad sense.
There are a number of processes for which one might be looking for a signature of this process. Such signature can take the form of various statistics. For example, a historically important statistics is Tajima's D defined as
$$D = \frac{\Pi - S/a}{SE}$$
, where $\Pi$ is the average number of pairwise differences between two randomly sampled haplotypes, $S$ is the number of segregating sites, $a=\sum_i^{k-1}1/i$, where $k$ is the number of sampled haplotypes. $SE$ is the standard error for the numerator which takes a complicated formulation (have a look at Tajima 1989 for more information).