I just read this in an answer:
As your last of piece of unfinished business, make a point of thanking the person who referred you to Company B. Yes, you decided to stay with A but thanks to that person, you got to know Company B and Company B is indeed a good company :)
Wouldn't that cause problems for the referring person? Maybe they already forgot how you got there, but this will remind them that Joe brought in this unreliable person that cancels a contract.