What is the best way to decline an offer, when you promised the hiring manager that you will join a certain date?
He said "I just want to make sure that you won't bail out along the way, and we're holding this position for you".
To which I replied "Don't worry, I give my word, I will join you".
But now circumstances have changed, and I can't join them. What is the best way to communicate him about this? Will bridge get burned?
As with every other time this question is asked here:
Will bridge get burned?
Yes. You gave your word, you are breaking it.
If there's actually a reason you can't join (e.g. a medical issue which you have to deal with) then explain that and the bridge might only be slightly charred.
If it's actually just that you're choosing not to join (e.g. because you have a different offer), then accept that you are choosing to break your word and accept the consequences of that.
What is the best way to communicate him about this?
Let them know as soon as possible, that you changed your mind, apologize, and move on. You can provide as much detail / background/ reasoning as you want, that's not going to change the fact that you're not joining them and they need to find someone else for that position. Do not delay any further to communicate that, and be polite but straightforward about it.
Will bridge get burned?
Definitely.