I set up a villager breeder and set down 4 beds with 2 villagers in the middle and a carrot farmer supplying them with carrots. The first time both of those villagers bred the breeder worked perfectly fine. After that, however, the gray angry particles popped up every time they would try to breed (they would engage in breeding at a normal pace, just could not actually produce a baby). I even killed the one villager they produced in case it took a bed and even then, the two villagers would not breed anymore. Is there something I have done wrong? This seems odd considering they were only able to breed once and even by eliminating their offspring the villagers could not breed again.
Edit: version is 1.16.4