By definition the work done by an external force $\vec F_{\rm external}$ in taking unit positive charge from position $a$ to position $b$ is the difference in potential between position $a$ and position $b$.
$V_{\rm b}-V_{\rm a}=\int_a^b \vec F_{\rm external} \cdot d\vec l$.
So if the unit positive charge is moving towards another positive charge the external force acting on the unit positive charge has to be directed towards the other positive charge which means that overall the dot product of the external force and the displacement of the external force is positive as they are both in the same direction.
This means that the external force does positive work and in moving towards the other charge the potential increases.
Going away from the other positive charge the dot product of the external force and the displacement of the external force is negative as they are in the opposite directions.
This means that the external force does negative work and in moving away from the other charge the potential decreases.
Now the electric field due to the other charge $\vec E$ is producing a force $\vec E$ on the unit positive charge.
When the unit positive charge moves towards the other charge the work done by force $\vec E$ is negative because the direction of that force is away from the other charge and the displacement of that force is towards the other charge.
When using force $\vec E$ to define the change in potential to be consistent with the external force definition it is necessary to introduce a negative sign so that $-\int_a^b \vec E \cdot d\vec l$ is a positive quantity and again there is an increase in potential as one gets closer to the other positive charge.
So you have an alternative definition for the difference of potential of $V_{\rm b}-V_{\rm a}=-\int_a^b \vec E \cdot d\vec l$ which is minus the work done by the electric field when a unit positive charge is moved between the two positions.
Going away from the other positive charge the force $\vec E$ and the displacement are in the same direction so the work done by force $\vec E$ is positive.
So minus the work done by the electric field is negative which means that, as expected, the potential decreases.