Quantum-mechanical object that captures some of the characteristics of particles associated to interactions with themselves and with other particles, such as the change in its mass as a function of its energy. Particularly useful in quantum field theory, either for relativistic particle physics or condensed matter physics. Although it also makes sense as a classical concept, it mostly has fallen into disuse nowadays.
Background.
In general terms, the self-energy $\Pi_{ij}(x)$ of a field $\phi^i(x)$ is defined as the inverse of its two-point function: $$ \int_{\mathbb R^d}\Pi_{ij}(x-y)G^{jk}(y-z)\ \mathrm dy\equiv\delta(x-z)\delta_i^k $$ with $G^{jk}(y-z)\equiv \langle \phi^i(y)\phi^j(z)\rangle$.
In perturbation theory, $\Pi(x)$ can be calculated by summing all one-particle-irreducible Feynman diagrams with two external legs. For this reason, $\Pi(x)$ plays an essential role in the study of the renormalisability of the theory.
Among other things, this function carries the information about the masses of the particles described by $\phi$. In particular, the Fourier transform $\tilde\Pi(p)$ vanishes at $p^2=\mu^2$ if and only if $\phi(x)$ creates a particle of mass $\mu$. This means that, for example, the system is gapless if $\tilde\Pi(p)=cp^2+\mathcal O(p^4)$ for some constant $c$.
For more details see the Wikipedia page self-energy.
