Assuming a domain of the natural numbers, if $\frac{a}{b}$ is a non-unit fraction between $0$ and $1$ in lowest terms, and $\frac{1}{n}$ is the largest unit fraction less than $\frac{a}{b}$, and $\frac{a'}{b'} = \frac{a}{b} - \frac{1}{n}$, then is it true that $0 < a' < a$?
Essentially this process is a single application of the greedy algorithm for egyptian fractions.
$$\frac{1}{n} \leq \frac{a}{b} < \frac{1}{n-1} \Rightarrow b \leq na < a+b \Rightarrow 0 \leq na-b < a $$
As $$ \frac{na-b}{b}=\frac{a'}{b'} $$ and the second fraction is reduced we have $$a' \leq na-b <a$$
Also, since $ \frac{na-b}{b}\geq 0 $ it follows that $a' \geq 0$
We have $\frac{a'}{b'}=\frac{na-b}{nb}$ with $na-b>0$ and know that $na-b\le a$ with equality if $(n-a)a=b$, i.e., $\frac ab=\frac1{n-1}$, i.e., $a=1$. (And indeed, your method applied to $\frac 1b$ results in picking $n=b+1$, making $\frac{a'}{b'}=\frac1{b^2+b}$)
