In this answer I address with various levels of mathematical sophistication/rigour why the minus sign is already taken into account by the limits of integration. If you want to completely avoid such confusions, note that line integrals are defined along paths. So, we should denote a line integral of a vector field along a curve as
\begin{align}
\int_{\gamma}\mathbf{E}\cdot\,d\mathbf{l},
\end{align}
where $\gamma:(t_1,t_2)\to\Bbb{R}^3$ is a smooth curve (actually $C^1$ is enough). Here, we allow $-\infty\leq t_1<t_2\leq \infty$, and if we allow for infinite values, then we should of course ensure that everything is convergent. Recall that by definition, this is equal to
\begin{align}
\int_{\gamma}\mathbf{E}\cdot d\mathbf{l}:=\int_{t_1}^{t_2}\left\langle\mathbf{E}(\gamma(t)), \gamma'(t)\right\rangle\,dt,
\end{align}
where $\langle\cdot,\cdot\rangle$ refers to inner/dot product. The parametrization of $\gamma$ contains all the information about directions.
For example, suppose we want the path $\gamma$ to "start at infinity and end at point $\mathbf{r}$" (for convenience assume $\mathbf{r}\neq 0$). Then, a radial parametrization of this is given by $\gamma:(0,1]\to\Bbb{R}^3$, $\gamma(t)=\frac{1}{t}\mathbf{r}$. Note that as $t\to 0^+$, we have $\|\gamma(t)\|\to \infty$, and at $t=1$, we have $\gamma(1)=\mathbf{r}$. So, this perfectly captures our idea of "starting at infinity and ending at $\mathbf{r}$". As a technical remark, note that there is no point $\infty\in\Bbb{R}^3$, which is why I only said $\lim\limits_{t\to 0^+}\|\gamma(t)\|=\infty$, I didn't say $\gamma(0)=\infty$, because that would be a nonsensical statement. Ok, now note that
\begin{align}
\gamma'(t)&=-\frac{1}{t^2}\mathbf{r}.
\end{align}
Notice how there is a minus sign here. This minus sign is precisely capturing the direction of traversal along the path.
Ok, the case of moving a particle from infinity is slightly iffy to deal with mathematically because the parametrization isn't so obvious. Let us deal with another case. Say we fix $0\leq r_1<r_2<\infty$. Let $\mathbf{u}$ be a unit vector along some direction, let $\mathbf{r}_1=r_1\mathbf{u}$ and let $\mathbf{r}_2=r_2\mathbf{u}$. Suppose we wish to travel from point $\mathbf{r}_2$ to $\mathbf{r}_1$ (i.e decrease in radius). One way of parametrizing the path is $\gamma:[r_1,r_2]\to\Bbb{R}^3$, $\gamma(t)= (r_2+r_1-t)\mathbf{u}$. Notice that we have $\gamma(r_1)=\mathbf{r}_2$ and $\gamma(r_2)=\mathbf{r}_1$, i.e we start at the point $\mathbf{r}_2$ and end at the point $\mathbf{r}_1$, as desired. Now, notice that
\begin{align}
\gamma'(t)&=-\mathbf{u}
\end{align}
This minus sign is once again encapsulating the direction of travel (from a larger radius to a smaller radius). Hence, the integral of a vector field $\mathbf{E}$ in this case would be
\begin{align}
\int_{\gamma}\mathbf{E}\cdot \,d\mathbf{l}&:=\int_{r_1}^{r_2}\langle\mathbf{E}(\gamma(t)),\gamma'(t)\rangle\,dt\\
&=\int_{r_1}^{r_2}\langle\mathbf{E}((r_1+r_2-t)\mathbf{u}, \color{red}{-}\mathbf{u})\rangle\,dt
\end{align}
The red minus sign takes the direction into account. Also, one can switch the order of limits of integration at the cost of a minus sign and write this as
\begin{align}
\int_{r_2}^{r_1}\langle \mathbf{E}((r_1+r_2-t)\mathbf{u}), \mathbf{u}\rangle\,dt.
\end{align}
The usual physics way of writing down the calculations is to just avoid explicitly writing out the parametrized curve $\gamma$; rather we simply indicate in the limits of integration the starting and ending points, such as $\int_{\text{start}}^{\text{end}}$ (this is only ok in the case of conservative force fields, because otherwise line integrals are path-dependent), and once you write this out as a one-dimensional integral, the signs are already captured by the placement of the limits of integration (remember that for Riemann integrals, if $b<a$ then we define the symbol $\int_a^b$ to mean $-\int_b^a$... this is the minus sign which you're overlooking). I'd suggest reading the link above to see a simple illustration of this in one-dimension.
