Most likely this issue is related to signer (and counter-signer, if available) certificate validation, which requires chain building and revocation checking by downloading required objects (certificates, CRLs, OCSP). Network retireval may cause such delays. In addition, there might be issue with CDP/AIA extension URLs which contain inaccessible URLs.
What I would suggest to do: extract signer certificate to a file and run certutil command against this file:
certutil -verify -urlfetch path\signer.cer
and examine output. Pay attention to URL retrieval information. If presented, repeat this command against counter-signer (timestamp) certificates. If necessary, post certutil output here to get additional help.