According to Bulbapedia:
Pokémon have a set capture rate that is added to depending on how many moves a player has left at the end of a battle (Example: 50% plus 5% per move remaining.)
Timed levels have a per-3-seconds-group bonus that is added for each time 3 seconds can be subtracted from the time remaining before it becomes zero or negative.
If capture fails and the player has at least 2500 coins, they are offered the choice of buying a Great Ball that has twice the probability of capture of the Poké Ball used for free.
So it's not calculated, it's a fixed value. The only thing you can do to increase the Catchability is end a challenge with a good amount of left moves or time. And eventually use a Great Ball.
Note that Moves+5 and Time+10 do not increase capture rate for those bonus moves, so you can't use these items to make it easier to catch a Pokémon.
If you end the stage with 5 or fewer moves, the Pokémon will have the same capture rate as if you beat it with 0 moves remaining.
EDIT:
The latest update introduced a new feature called Super Catch Power:
After successfully completing a stage, if the player fails to catch a Pokémon with their first Poké Ball (before using a Great Ball), they have a chance of being offered a "Super Catch Power". This Super Catch Power provides a somewhat random boost on top of the Great Ball's boost (the player still uses a Great Ball, and can use the same number of Great Balls as they could without the "Super Catch Power")