The prompt window which appears on windows authentication is highly likely because of 401 Access Denied
While I am not sure about some mandatory questions to be able to tell the exact same reason like
A- what authentication provider is being used here?
B- Do you need Kerberos or not? are you using a custom identity or not?
I believe using Kerberos authentication as a provider with Windows Authentication is the best option and to do it please ensure you did all the below steps carefully in a right way
1- Create a custom account on Active Directory, delegate this account from the delegate tab on AD Properties to ensure it can user Kerberos
2- use this custom identity as an app pool custom identity user, go to application pools, choose your app pool, right click, custom identity and set the user you just created.
3- go to application authentication tab, disable all authentication providers including impersonation (not only anonymous) except windows authentication
4- right click on Windows Authentication, choose providers, and choose "Negotiate/Kerberos" as primary provider below it "Negotiate"
Authentication Providers
5- Set Service principle name, open CMD, set SPN to the service account such that if service account is "lab\testuser" and server domain is "server1A" and its FQDN (Fully Qualified Domain Name) is "server1A.test.com"
type the below command:
setspn -s server1A lab\testuser
setspn -s server1.test.com testuser
it is really important to clear Kerberos cache ticket as well because
manytimes you will make changes and you won't see it took any effect
except after clearing the cache, so you need to use [KLIST tool][3]
to clear it by typing the command klist purge
then you need to clear the DNS cache, restart IIS by typing the commands below on CMD "Run as admin"
ipconfig/flushdns
iisreset
Good Luck, please let me know if you have any further questions, if you need clarification for anything