As you are traveling around after Level 4, you will soon notice a ghost following you. Although this will probably scare you at first, he will tell you that all he wants is to be taken back to his old house by the sea. You will have to do what he wants, because you cannot enter Level 5 until you do. His house is located just to the east of where you talked to Marin by the seashore. Take the ghost to his house and let him look around inside. Soon, he will become overwhelmed with memories and ask you to take him back to his grave. His grave is not located in the main cemetery, but it is the single headstone just to the west of the Witch’s Hut, which is just east of the Mysterious Woods. Once you return him to his grave, he will tell you that under one of the jars in his house is a gift for helping him. So go back to his house and look under all the pots. Under one of them, you will find your reward, a Secret Seashell!
Source Here
EDIT
You can take the ghost straight to his house, then straight to his grave.
An example would of this is walk-through to about 5 minutes in.