Food flavors and odors are comprised of complex mixtures of hundreds of chemicals, which act as stimuli to the human sensory organs, for example, human nose. The odor of food may be attributed to the odor of a single chemical or an integrated response by nose to a mixture of chemicals. This reference to a single chemical or a mixture of chemicals allows trained analysts to develop a common terminology in describing odor, one of which called odor thresholds. Threshold is defined as the degree of stimulation of a nerve or nerve center that just produces a response. If the concentration of the odorous chemical (usually given in parts per million or ppm) is below this point, it will be judged not odorous. Hence, accepted definition of threshold is the minimum
identifiable odor (MIO) or recognition threshold. MIO of some chemicals is very high while some is extremely low. The latter chemicals are the ones you said, "hard to get rid of." For example, the oder of dimethylformamide is described as 'fishy' and pungent, but you won't smell it if its concentration in air (or on your skin) is below $\pu{100 ppm}$. On the other hand, tar-like, pungent smell of p-cresol can be detected if its concentration in air (or on your skin) is greater than $\pu{0.001 ppm}$.
Now considering the fish smell, it was described short and sweet in following website from American Society for Nutrition: Read
Accordingly, it says:
Water in the open ocean is about 3% salt by weight, but the optimal levels of dissolved minerals inside an animal cell is less than 1%. In order to maintain fluid balance, ocean creatures must fill their cells with amino acids and amines to counter the saltiness of seawater. Ocean fish tend to rely on trimethylamine oxide (TMAO) for this purpose.
The problem is that when fish are killed, bacteria and fish enzymes convert TMAO into trimethylamine (TMA), which gives off the characteristic “fishy” odor.
Now, it is nice to know MIO of trimethylamine (TMA) is $\pu{0.00021 ppm}$ or $\pu{0.21 ppb}$ (parts per billion). That's why it is so "hard to get rid of" fishy smell whenever you touch or cook dead fish. But the good news is TMA is a organic base, which would react with any acid to form water soluble ammonium salts. Citric acid from lemon (or lime) or acetic acid from vinegar would do the job.