So energy comes to earth from sun in the form of EM waves. Some of it is reflected back but some of it remains on earth and is used by plants to create food and some is used in the atmosphere creating storms. But how is this energy sent back to space?
If it is not sent back, then does it mean that the total energy on earth as a whole is continuously increasing?
There are mainly two mechanisms that send radiation back into space: reflection and emission. In the first case, electromagnetic waves coming from the sun are simply reflected, with the same high frequency. This is due mainly to clouds, which in this respect contribute to cooling the earth's temperature. The fraction of the incoming light that is reflected by a planet (or any other object) is called albedo. For the earth, it is about $0.3$, which means that about $30\%$ percent of all the energy from the sun is already reflected into space by reflection (this is a rough estimate, different parts of the earth have different albedos: oceans and forests have a lower one, poles a higher one and so on).
What is left can be absorbed and then re-emitted with a lower frequency as thermal radiation (typically in the infrared). As far as emission is concerned, we have to take into account the effect of the atmosphere (for example, the ozone layer subtracts energy, absorbing the higher frequencies), of the surface and of clouds (apart from reflection, they also absorb the radiation emitted by the surface; part of it is emitted again toward the surface and part out into space - the mechanism is complicated and typically depends on the difference in temperatures between the top and the bottom of a cloud, the type of particles in the clouds and so on). The net effect is that the atmosphere raises temperatures on Earth (without the atmosphere, the average temperature would be $-19$ degrees Celsius or so). But in the long run (apart from what Ron mentioned), all the energy is re-emitted into space. This may seem paradoxical, but as pointed out for example by R.Penrose, what really matters for life is that the sun is a hot spot in an otherwise dark sky (it is a source of low-entropy energy: yellow energetic photons are absorbed by the earth and emitted in the infrared in larger numbers; this means more degrees of freedom, more volume occupied in phase space or more entropy).
It is all sent back eventually, mostly at wavelengths comparable to infrared. The only residual energy stored is where there is an increasing amount of organic material (produced from solar energy) deposited in riverbeds and seabeds, which is greater than the material breaking down at the bottommost layer of the bottom organic layers. If the organic layers at the bottom of the ocean are in a steady state, neither increasing in total mass or decreasing in total mass, the amount breaking down at the bottom and releasing heat is equal to the new material placed on top.
The reason is what you say--- the Earth doesn't have anywhere to store the energy in steady state. The Earth radiates slightly more energy than it recieves, since there are radioactive materials in the core.
