I have a full-time work contract for 40 hours per week. However, the working time is not actively controlled, but completely trust based with a 4 hour core time. I was told informally that it's not really important to be in the office during the core time as long as I'm available for meetings, etc. I have also the (un?)fortunate situation that neither my direct manager nor his manager are on site, so there's also no indirect control for me.
I've noticed that I can do my work with 4-6 hours of daily effort. My direct manager likes my work and thinks I'm doing well. Hence I'm inclined to go home after around 6 hours, technically not putting in the 40 hours. Otoh, my contract is trust based and I do the work I'm assigned (and do it well).
Actually, I wouldn't have a problem with this, but I fear that my colleagues here in the local office might get upset. However, staying in the office for the full amount of time every day is very frustrating for me. I should also note, that I strongly dislike being in the office and try to get my time and work here over as fast as possible. I don't go to launch like everyone else for example (which they do during the core time, what a joke). Theoretically, there's also the option to do some work from home, but I was denied that for arbitrary reasons.
So, how could I deal with this, without risking to get fired or upsetting my colleagues? Those colleagues are not my direct colleagues, more like office mates. They have a different manager and different work. I'm the only one from my "team" here.
As suggested in the comments, here's a bit more information to clarify my situation:
The current situation is a complete mess. I've talked to my manager about the low work load, that I'm unsatisfied and under-challenged more than once. Nothing changed. He keeps thinking that I have enough work to do. I've come to the point the embrace the situation and make the best of it, i.e. do my work and leave asap.
To be more general, if my work time is trust based, why is it not ok to leave early when I have done all my work for the day?
Addendum: There are many good answers, so I have a hard time to decide which one to accept. I'd love to accept them all, because every answer has it's one unique facet to it, providing a new insight. There are also some very good tips and discussions in the comments, so make sure to check them out as well. I have to make up my mind and decide which answer comes closest to what I think is the best answer. Until then, thank you very much to everybody who contributed or still wants to contribute!