I disagree with the premise of your question that energy is a "simply mathematical object". Indeed, the fact that it can have density and flux is one the reasons why it ought to be considered a real, tangible thing instead of just a bookkeeping device. While energy in on all its forms is a property of something else, being a property doesn't reduce a physical thing to being simply a mathematical object.
As dmckee said, the idea is that when you have a field (any field, not just the EM field) you can meaningfully discuss the energy at a particular point. If a field can store energy, then it makes sense that different sized "chunks" of the field will store different amounts of energy. Think, for example, about electromagnetic waves that can deliver power to a receiver. A big box that contains many wave periods will obviously have more energy inside it than a small box that contains very few wave periods, since the wave delivers a certain amount of energy per period. There's nothing special about waves vs. other field configurations, so the same reasoning applies to any form of the electromagnetic, or any other, field.
Well, if it's meaningful to talk about the total field energy contained in some box then its meaningful to compute the average density of energy by dividing out the box volume. If I take the limit of smaller and smaller boxes around a point, this average density approaches the density at that point. We can go through the same sort of reasoning with momentum density (which is closely related to the energy flux).
The fact that, as you observed, energy appears to flow like a fluid is because conservation of energy is actually a much stronger statement than you're imagining. You're used to talking about energy conservation in global terms: the change in the energy of a system is equal to the work done on the system by its environment minus the work done on the environment by the system. In field terms, this means that if you draw any sized box, then the change in the total energy in the box equals the total flux going through the sides of the box. By making the box arbitrarily small, we get (using the divergence theorem) a local statement of conservation of energy: if $\rho(\vec{x},t)$ is energy density and $j(\vec{x},t)$ is the energy flux then $\frac{\partial\rho(\vec{x},t)}{\partial t} + \nabla \cdot j(\vec{x},t) = 0$. This is a very powerful statement and is more fundamental than the global statement. For instance, in general relativity local conservation of energy still holds but, because of the complications introduced by curvature, the global statement fails.
