You already identified one potential drawback - if you later apply to another job that pays less, people will wonder just why you would leave a highly paid position, unless there is an obvious reason (like your spouse getting a job somewhere far away and you needing to take what is offered there).
High paying jobs usually come with high requirements and expectations. If expectations are higher than for a low-paying job, you can more easily disappoint the people that pay you really well.
Next, when a manager needs to trim x% from his payroll, he might be more tempted to fire one expensive employee than two cheaper ones, so your high paying job may be less secure than a low paying one.
One other thing, of course, is that you can easily get used to the high salary, either in terms of lifestyle inflation (if you spend a lot) or in terms of peace of mind (if you save a lot). Either one can restrict the possible jobs you could accept later on. However, this really is a kind of first world problem. "Help, my current job pays so well that I can't find a competitive open position elsewhere!" Lots of people would love to have this kind of problem.