Ideally all parts of the brake system that hold the cable housing should be fixed and immovable. There are two problems that can occur if this is not the case:
Flexing or movement when the brake is applied and there is tension between the cable and housing: This reduces the force that can be applied at the brake caliper and is obviously a bad thing.
Movement of the housing as the brakes are applied and slack in the cable is taken up. The brake lever has to be moved too far before the brake pads touch the wheel rim.
Sounds like you have the second problem. When the brakes are released the springy housing is moving the brake hanger causing you to have too much slack in the system.
The fix is is to get the cable housing under control so that it cannot flex and move around. The housing may be a little too long or not anchored the the frame properly. You can try fixing the brake housing to the frame or seat post with a zip tie to hold it in place. A zip tie around the seat tube end if the hanger to make the housing enter into it straight might work also.
Ideally, there should be a little tension in the brake cable when the lever is released so that the housing is held in line. This tension is supplied by the return springs of the calipers. Check the calipers to see if the return spring force is too weak. There are typically adjustment screws on the calipers or a array of holes on the brake bosses that the end of the spring fits into for adjusting spring tension.
Update after reply from OP in comment: 'I have shifter-type compressionless housing ...'
Shifter cable housing should NOT be used for brakes. The forces generated by brakes are much higher than those generated by shifters and derailleurs and the housing may fail.
See this answer.