Is the person holding and moving the book losing energy, and why?
Person holding and moving the book is actually losing energy all the time. The reason is that gravitational potential energy cannot be converted back to chemical energy stored in our muscles. Even when our body is seamlessly stationary, our muscles are constantly under motion on a very (very) small scale. This is the main reason why you cannot hold a book forever.
Although we are often trying to relate different concepts in physics with our intuition, which can help sometimes, you should not do this with work as defined in physics and effort done by our muscles. It will often lead to wrong conclusions.
The work energy theorem tells us that change in kinetic energy equals total work done on the object
$$\Delta K = W$$
The total work $W$ can be written as a sum of work done by gravitational force and work done by other forces (such as a person lifting a book)
$$\Delta K = W_g + W_\text{other} \tag 1$$
When you lift the book from $y_1$ to $y_2$, where positive $y$ direction points upwards and $y_1 < y_2$, the gravitational force is doing negative work on the book. Since book starts and ends with zero kinetic energy, the Eq. (1) for this case becomes
$$-mg(y_2-y_1) + W_\text{other} = 0 \qquad \rightarrow \qquad W_\text{other} = mg(y_2 - y_1) > 0$$
The work done by the person on the book $W_\text{other}$ is positive since $y_2 > y_1$, and the work done by the book on the person is negative which follows from the Newton's third law of motion. This is somehow intuitive.
When you put the book back down from $y_2$ to $y_1$, the kinetic energy difference is again zero and the work done by the person on the book is
$$mg(y_2-y_1) + W_\text{other} = 0 \qquad \rightarrow \qquad W_\text{other} = -mg(y_2-y_1) < 0$$
The work $W_\text{other}$ is now negative, which means the work done by the book on the person is positive. This is what physics tells us, and it is correct. What is not intuitive is the question "Where does this gained energy go?".
Major problem here is that we are trying to understand the work via effort done by our muscles. This is not equivalent and should be avoided. Although the book does positive work on the person when it goes down, the person still has to put some effort in preventing the book from free falling. This is done by constant contractions of our muscles which dissipate energy, i.e. they convert it to heat and maybe some other forms of energy.
When lifting the book, the effort done by our muscles converts to increase in gravitational potential energy of the book-Earth system. When putting the book down, the effort done by our muscles converts to decrease in gravitational potential energy of the book-Earth system. In both cases our muscles do effort, because gravitational potential energy is not converted back to chemical energy in our body (muscles).
