$$
r\ddot \theta+2\dot r \dot \theta = 0 \implies 2\frac{\dot r}{r} + \frac{\ddot\theta}{\dot\theta} = 0 \implies \ln(r^2\dot\theta) = C \implies r^2\dot\theta = C \tag{1}
$$
By the condition $F=mr(0) \dot \theta(0)^2$ and the equation
$$
\ddot r-r\dot \theta^2+\frac{F}{m}=0 \tag{2}
$$
at $t= 0$ you can see that $\ddot r(0) = 0$. Substituting $(1)$ in $(2)$ to eliminate $\dot \theta$ yields
$$
\ddot r -C^2\frac{1}{r^3}+\frac{F}{m}=0 \tag{3}
$$
Now taking all derivatives of $(3)$ at $t = 0$, knowing that $\ddot r(0) = \dot r(0) = 0$, should show recursively that $\frac{d^n}{dt^n}r(0) = 0, n\ge 1$, so the Taylor expansion of $r(t)$, (if it has one,) will have only one nonvanishing term, namely the constant one $r(0) \ne 0$, (it is essential for this method that this term to be nonzero).
This result and the same idea can be use to find $\frac{d^n}{dt^n}\theta(0) = 0, n \ge 2$ from the equation ($1$) rewritten as
$$
\dot \theta = \frac{C}{r^2}
$$
Uniqueness
Set $\dot r = u$ and $\dot \theta = v$, then
\begin{align}
\dot r &= u \\
\dot \theta &= v \\
\dot u &= rv^2-\frac{F}{m} \\
\dot v &= -2\frac{u}{r} v \\
\end{align}
The function
$$
f:\mathbb R^4\backslash\{(0,\theta,u,v)\} \to \mathbb R^4, \qquad (r,\theta,u,v) \mapsto \left( u,v,rv^2-\frac{F}{m},-2\frac{u}{r} v \right)
$$
is locally Lipschitz in every neighborhood with $r \ne 0$ because it is differentiable, i.e., it has continuous partial derivatives. So one can use the on such neighborhoods. So the uniqueness of the solution follows from, as the "mathcounterexamples.ne" noted, the Picard-Lindelöf Theorem.
For my solution above I didn't like invoking the Taylor series, but I couldn't find other ways in the time being
– Physor Oct 23 '20 at 15:04