As you can see Introduction to the Monte Carlo Method, by Andrea Pelissetto, on pag. 143 (section 6), detailed balance is satisfied as long as the acceptance probability for a move is
\begin{equation}\tag{6.143}
\frac{F(z)}{F(1/z)} = z
\end{equation}
where
$$
z = \frac{\pi_j p_{ji}}{\pi_ip_{ij}}
$$
and the $\pi$ denote the equilibrium probability of a given state and the $p$ the probability of proposing a given transition.
As long as this condition is satisfied, the Metropolis algorithm will work. Of course, since $F(z)$ is a probability, it has to be less than 1:
$$
F(z) < 1\,.$$
Moreover, since the probability to undergo the reverse transition must also be less than one, we have
$$
F(1/z) < 1
$$
and substituting this equation in the (6.143), we get
$$
F(z) < z\,.
$$
Therefore, it must be
\begin{equation}\tag{1}
F \leq \min(1,z)
\end{equation}
and if you want to maximise the acceptance probability (which you probably do, so that your system evolves as fast as possible), you can set
$$
F(z) = \min(1,z)\,.
$$
This leads us to finally answer your questions:
Can I choose a different F(z)? Yes, as long as the condition 1 is satisfied. E.g., $1/(1+z)$ is perfectly valid. However, your system will be moving slower than it should.
Do I have to use the Gibbs distribution? No. You can use the metropolis algorithm to sample any distribution you like, however, you are likely to be dealing with systems distributed according to the Gibbs distribution.
What if I introduce a third option, apart accepting or rejecting the function? Then it's not the Metropolis algorithm any more, but it's probably something more similar to the heath bath algorithm (same file and section as before (section 6))
