The short answer is, it's there to let you adjust the picture, with the intent that it looks pleasing to your eye. Therefore, if you're happy with the way the picture looks, the gamma's probably fine. In gaming, extreme adjustments to gamma might expose or hide similarly colored items, which might change the game slightly, but otherwise it's a personal preference.
The longer answer follows.
Gamma's one of those tricky things that computers do to account for the fact that humans aren't as linear/digital as computers are. Computers can represent way, way more colors than humans are capable of telling apart. We humans have a tough time telling shades of gray apart - especially when they are very close to white or black. We're far more sensitive to shades of gray in the middle of the range between white and black. We use tricks like gamma encoding/decoding to do our best to take the zillions of possible colors and make the small amount we can efficiently represent have the most impact.
Gamma encoding attempts to capture as many different perceptible colors, while lumping together colors we can't tell apart. This is done to reduce the number of colors that the computer needs to store internally, while maximizing how many different colors can be shown. Think of it like a form of compression.
So, an artist creates an image as an art asset for a game. He uses particular colors to make this happen. The computer encodes these colors into a file with a particular gamma. When you boot the game up, your monitor or TV interprets these colors and creates an image out of them. This process of converting the digital signal back to something your eyes can see is the gamma decoding step.
However, not all displays are created equal, and there's a decent amount of variance between even different displays of the same model from the same vendor. Thus, if we want the image to look the same to everyone experiencing the same image, we need to calibrate and adjust the decoding process to account for these differences.
The simplest way is to show a pattern like the one you describe. If you follow their directions, you'll typically end up with a pretty decent approximation of the gamma that was used to encode the art assets used in the game.
If you adjust your gamma outside of the directions, the art isn't going to appear as the artist intended it. However, it might give you certain advantages by expanding the color range in an area where maybe the artist didn't intend for you to be able to distinguish.
It all comes down to how you prefer to experience the game. If the gamma's too low, and too many dark colors are lumped together, then the game might be hard to play. If the gamma's too high, you might miss lightly colored objects that are supposed to be differently colored, but which aren't because the gamma adjustment is lumping them together.
My advice? Unless you're a professional concerned with color, print, video, etc... just treat it like a preference. The adjustment is there to give you the chance to account for differences in your display, your eyes, the lighting conditions in the room, etc. Put it where you like it, and enjoy the art. :)
