Your job is whatever your boss decides it is. So yes he can assign you any random thing he wants to. You mention some are in the name of career development so those are things he thinks you need to work on to be qualified for promotion. They aren't random things, they are important. Just because they aren't what you wanted to do doesn't make them random. Often junior people have no idea of the skills they will need at a more senior level.
Further, sometimes he just has tasks that no one wants to do. Those are going to be assigned to people as well. Sometimes that person is going to be you.
It is unrealistic to expect that you will never have work assigned that is not what you personally are interested in doing.
As far as colleagues, you can sometimes say no, but it is often not a good idea from a teamwork standpoint. Discuss with your boss the types of requests you get from colleagues and come up with a plan for what you should say yes to, what you should turn down, what your priorities are and how to handle turning down the request. The most common way to turn down a colleague's request is to point to your priorities and tell them that if they need your time, then they need to talk to your boss. It is his job to juggle priorities, so let him do it.
As you get more experienced, you will find that doing some of those tasks that take only a couple of minutes is well worth your time. People will be more cooperative when you need help.
