How Do I Fully Flush Cached Redirects From Safari?

Question

I have a device with a web-based control panel, and accidentally set it to redirect all http pages to https, even though some don't work over https. Although I've since corrected this, Safari seems to have memorised the redirect and is refusing to forget it, instead constantly attempting to redirect me to the invalid https address.

I've already closed Safari, cleared ~/Library/Caches/com.apple.Safari/ and ~/Library/Cookies/HSTS.plist but it still seems to be remembering the redirect when I re-open it.

Where else could Safari be storing this information? I can access the correct page via Firefox or Chrome, so it may not be a system wide service, or if it is it's not one that the other browsers use.

Unfortunately because the web-panel is provided by a device I don't believe I can adjust headers or setup a redirect back to the correct URL, which seem to be options offered in other similar questions, so I really need to find out where this data is being stored so I can destroy it with fire.

See also http://apple.stackexchange.com/questions/153503/safari-8-not-clearing-redirect-data — Paul D. Waite, Nov 03 '16 at 12:28
Have you tried trashing/moving aside your ~/Library/Safari folder and seeing if that fixes the problem? If it does, you can experiment with items inside the folder until you find the culprit file. — interestinglythere, Dec 31 '16 at 14:03
How did you set the redirect? With an extension or is there a setting in Safari for that? — owlswipe, Dec 31 '16 at 15:55
Does the redirect still happen with a private browsing window? — AllInOne, Jan 04 '17 at 20:39
@AllInOne interesting idea, but unfortunately it still happens under private browsing. — Haravikk, Jan 05 '17 at 10:20
@owlswipe the redirect was the result of mistakenly enabling HSTS on the server (which tells the browser to always use https), however not all of the services on the device support https, hence my problem, as Safari keeps trying to switch to https and then fails because it can't connect. I disabled HSTS, and removed the ~/Library/Cookies/HSTS.plist file that's supposed to handle it, but it still happens in Safari. — Haravikk, Jan 05 '17 at 10:22
Common for Safari & Chrome: 1) Clearing the domain-specific storage also reliably clears the cached HTTP redirections for that domain. But there are no tools to inspect/manipulate the cached HTTP redirections.
A similar thread on Google Chrome says: "It's unclear where Chrome stores the redirects."

Suspicion: Due to the possibly short redirection expiration date and/or performance requirements often solely in RAM, not persisted.

Also how long to cache HTTP redirects varies greatly across browsers — porg, Jan 27 '23 at 18:15

score 44 · Accepted Answer · answered Jun 26 '17 at 17:56

44

Based on quanta’s answer:

I wasn’t able to use launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist because I have System Integrity Protection enabled:

$ launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist
/System/Library/LaunchAgents/com.apple.nsurlstoraged.plist: Operation not permitted while System Integrity Protection is engaged

However I was able to work around it by doing the following:

killall nsurlstoraged (stops your user’s nsurlstoraged process; I actually ran sudo killall nsurlstoraged, but I suspect it isn’t necessary to stop the system’s nsurlstoraged as well, since the cache is in the user Library folder)
rm -f ~/Library/Cookies/HSTS.plist (deletes the HSTS cache)
launchctl start /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist (restarts nsurlstoraged)

answered Jun 26 '17 at 17:56

Grant Heaslip

556

I can't upvote this answer enough. It appears that at least on Sierra, simply removing the HSTS.plist file won't fix the issue because it will continue to be rebuilt. However, after killing nsurlstoraged and then removing the HSTS file—that did the trick! – nvahalik Aug 28 '17 at 16:52
1
Thanks very much, upvoted, but I did it like this.
1. Close Safari
2. Edit ~/Library/Cookies/HSTS.plist and remove the entry for the site I want on http
3. Restart computer
– Jason S Feb 03 '18 at 21:42
Yeah, restarting is the advice that all the other answers give you, but with 20 apps open it's much more convenient and faster to restart the nsurlstoraged process. Thanks @nvahalik ! – axello Apr 30 '18 at 21:38
4

Mojave update: the command rm -f ~/Library/Cookies/HSTS.plist will return Operation not permitted unless you have granted Full Disk Access to Terminal.app in System Preferences => Security & Privacy => Privacy. Otherwise, the solution worked perfectly! Thanks! – joehanna Mar 04 '19 at 06:36
@ nvahalik What’s going on seems even odder than the file being rebuilt; not even rm ~/Library/Cookies/HSTS.plist ; touch ~/Library/Cookies/HSTS.plist ; chmod guo-wrx ~/Library/Cookies/HSTS.plist helped me, but killall nsurlstoraged did. – Flash Sheridan Jun 17 '19 at 16:04
1

This doesn't appear to do anything in Big Sur. The killall command returns 'No matching processes belonging to you were found' – darylknight Feb 04 '21 at 13:30

score 9 · Answer 2 · answered Feb 16 '16 at 07:54

9

If you enable Develop menu in Safari preferences, you can clear cache from there (CMD+ALT+E).

Can you confirm that opening the device's control panel in Safari's Private window (or different web browser) works correctly?

answered Feb 16 '16 at 07:54

Filip Jurik

148

1

Unfortunately the develop menu option doesn't seem to clear the redirect, neither does closing Safari and manually deleting ~/Library/Caches/com.apple.Safari so the redirect must be getting stored somewhere else. HSTS was the feature I accidentally enabled but I've already deleted ~/Library/Cookies/HSTS.plist. – Haravikk Feb 17 '16 at 09:02
1

I can also confirm this answer doesn't fix it – malhal Aug 15 '17 at 22:02
1

This one worked for me – Matthew Cawley Oct 05 '18 at 11:19

score 8 · Answer 3 · answered Jun 01 '17 at 09:44

Based on @Haravikk's answer: https://apple.stackexchange.com/a/267783/62907

Anyone have any ideas which process is responsible for the ~/Library/Cookies/HSTS.plist file?

fs_usage can help:

❯❯❯❯ sudo fs_usage | grep HSTS
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000238   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000009   nsurlstorage
16:11:03  open              /Users/quanta/Library/Cookies/HSTS.plist                                         0.016268   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000008   nsurlstorage
16:11:03    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000003   nsurlstorage
16:11:03  access            /Users/quanta/Library/Cookies/HSTS.plist                                         0.000011   dbfseventsd
16:11:04  lstat64           /Users/quanta/Library/Cookies/HSTS.plist                                         0.000008   fseventsd
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000006   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000002   nsurlstorage
16:11:08  open              /Users/quanta/Library/Cookies/HSTS.plist                                         0.000144   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000002   nsurlstorage
16:11:08    HFS_update      /Users/quanta/Library/Cookies/HSTS.plist                                         0.000003   nsurlstorage
16:11:08  access            /Users/quanta/Library/Cookies/HSTS.plist                                         0.000021   dbfseventsd
16:11:09  lstat64           /Users/quanta/Library/Cookies/HSTS.plist                                         0.000042   fseventsd

So, we can:

launchctl unload /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist

then:

rm -f ~/Library/Cookies/HSTS.plist

and try again.

Thanks! This worked for me. I deleted HSTS.plist many times (closing/restarting Safari before and after) and it was always being recreated with exactly the same contents as before. Unloading nsurlstoraged first, then deleting the plist and restarting nsurlstoraged gave me a clean plist. — lucianf, Jun 20 '17 at 14:07
You could improve this by mentioning you need to quit and relaunch Safari for it to work. Also rather than delete the HSTS.plist I just deleted the problem domain key. — malhal, Aug 15 '17 at 22:06

luckydonald · Answer 4 · 2023-09-29T10:46:03.020

7

In Safari, Firefox and Chrome as well, all you have to do is open the developer sidebar, select the network tab, and disable caching.

Activate that, and the old permanent redirect should be ignored.

The greatest benefit is that you don't have to mess with files, you won't delete all HTST entries and loose the security benefits. Also, it works across browsers.

Older Safari

In older Safari, that's a crossed out tube thing, the blue one next to the trash bin logo.

edited Sep 29 '23 at 10:46

answered Jul 14 '19 at 13:51

luckydonald

173

While this is very useful to know, can you please confirm whether it functions as a permanent solution? i.e- if the cache is re-enabled, will the problem resurface, or does temporarily disabling it flush it? – Haravikk Jul 15 '19 at 11:45
1

@Haravikk in my tests it would not revert to using the permanent redirect when a new page could be loaded instead. Even after closing the development window, if that answers your question – luckydonald Jul 15 '19 at 18:25
1

This is by far the best answer. Disabling the cache makes the browser replace the cached response for the resource, so when you enable the cache again it will not redirect. – PEZ Jan 13 '23 at 17:22
in 'modern' safari (i.e. in Safari 17) it's just a checkbox called 'Disable Caches' in the Network tab, and it's not some weird icon anymore -- for me this worked great!!! thanks – MichielB Sep 21 '23 at 11:42
Just tested on Safari 16.6 and while the cache was not used while the flag is cleared, it was reused immediately after I checked the flag again – Alfonso Sep 28 '23 at 07:34

Haravikk · Answer 5 · 2017-01-07T23:37:30.413

So I've found a workaround to the problem, though this isn't a definitive answer to the actual question so I will not mark it as such until I can find more information.

It turns out that the file ~/Library/Cookies/HSTS.plist was indeed the source of the problem as I suspected, however deleting it from the affected user account doesn't work, even with Safari closed, as it is recreated after an unknown amount of time, complete with the offending entry that was forcing the invalid redirect.

So my solution was the following:

Make sure you have at least one other user account on your Mac (if not, create one).
Logout of the affected user account.
Login to a different user account (a guest account may not be sufficient, depending upon restrictions).
Find out the short-name of your affected user account; if you don't know then the best way to check is to look under System Preferences -> Users. Usually if will be the full name, lower-cased and with no spaces, so if your full name is "John Smith" then the short-name may be "johnsmith".
Open a window in Terminal, type su shortname replacing "shortname" with the short-name of the affected user-account. Hit enter and, when prompted, enter the password for the affected account.
Now type the next command rm ~/Library/Cookies/HSTS.plist and hit enter, this will delete the HSTS storage file.
Finally type exit, hit enter and close Terminal.

At this point you can now log back into the affected user account and the offending HSTS redirect should be gone for good.

Now, while this provides a usable workaround, I'd really like to know why deleting the HSTS.plist file from my affected account didn't work; the fact it is recreated means some background process is responsible for it, which means it should be possible to delete the file from the affected user account by simply stopping that process, deleting the file, then relaunching the process.

Anyone have any ideas which process is responsible for the ~/Library/Cookies/HSTS.plist file? Once we know that it should be possible to give a simpler fix to the problem.

score 3 · Answer 6 · answered Feb 16 '16 at 18:03

3

You will have good results if you use the command line to curl the device to make sure it's not doing the redirection. Safari doesn't really have an engine to rewrite addresses - especially if you go into private browsing to remove any history, cookies, etc...

If you're not sure you have cleaned your safari enough, you can also test by opening system preferences and making a clean/new user account on the Mac and test the site on a totally clean version of Safari after logging out of your normal user.

answered Feb 16 '16 at 18:03

bmike

235,889

There's definitely no redirect (the feature I'm trying to connect to doesn't support HTTPS at all, which is why enabling HSTS for the entire device was a terrible, terrible mistake); I can connect just fine from other user accounts and browsers, so there's something stored somewhere on my main account that is caching this :( – Haravikk Feb 17 '16 at 09:03
“Safari doesn't really have an engine to rewrite addresses” — I’ve currently got the same problem occurring in Safari with a website hosted on my laptop, and curl (along with Firefox, Chrome, and a Private Browsing window right there in Safari) on the same user account loads the site just fine. So it must be something to do with Safari itself. – Paul D. Waite Nov 03 '16 at 11:42

score 3 · Answer 7 · answered Aug 17 '17 at 20:30

3

After trying all of these solutions, what worked for me was:

Remove all instances of the domain from Safari's history
Quit Safari
Delete ~/Library/Cookies/HSTS.plist
Restart

answered Aug 17 '17 at 20:30

Edward Loveall

131

This is the correct answer to clear 301 redirect cache. None of the above work unless you also remove instances of the domain from Safari's history. – Dustin Feb 05 '20 at 17:32

AllInOne · Answer 8 · 2017-01-05T16:33:28.910

Here's an idea!

You say you can't undo the redirect by setting the server to redirect https requests back to http (as you don't have admin access to do so).

But what if you trick safari into connecting to a different server that offers this reverse redirect?

You could set this up in your local machine's /etc/hosts file.

For example let's say the current cached redirect is from http://example.com to https://example.com.

Now set up or identify a url that you can request on any server in the world that redirects from https back to http. Let's say that server has the address of https://redirecting.example.com.

Then look up the IP address of redirecting.example.com. In Terminal you can do like this:

host redirecting.example.com

You get a result something like this:

redirecting.example.com has address 69.69.69.69

Now open up your /etc/hosts file and add a new line that points requests for example.com at the ip address of redirecting.example.com, like so:

### point host example.com at the ip address of redirecting.example.com
69.69.69.69 example.com

Save your changes and clear your DNS cache in terminal like so:

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder; say DNS cache flushed

Then in Safari make a request for https://example.com the response should be a redirect back to http://example.com, at which point (fingers crossed) your Safari redirect from 6 months ago will be overwritten.

When done remove the line you added to your /etc/hosts file and flush your DNS cache again.

While it's a nice idea it doesn't solve the real problem; I'm not looking for workarounds, but rather I want to know where this redirect is being cached such that Safari continues to use it even though it is no longer valid (the server does not have HSTS enabled, I simply enabled it briefly by mistake). It must be stored somewhere, but I can't figure out where. — Haravikk, Jan 06 '17 at 18:29
This is not what I would call a workaround, as I expect it will solve the actual problem. It only works around the fact that you don't have control of the device. But I do hear you -- it would be nice to be able to clear the cached setting directly. Does Safari Technology Preview also exhibit the bad behavior? — AllInOne, Jan 06 '17 at 18:34
Unfortunately so; I don't think it's a problem with Safari itself as such, but rather some macOS service that it depends upon, as it does indeed appear that ~/Library/Cookies/HSTS.plist was the culprit, but deleting it from the affected account doesn't work (as it is recreated some time later, complete with bad redirect). Not sure what process is doing it though. — Haravikk, Jan 07 '17 at 13:45

score 2 · Answer 9 · edited Jul 14 '19 at 23:58

My two cents for new macOS Mojave 10.14 Beta (18A365a)

a) You cannot stop definitively nsurlstoraged, it relaunches in 2 seconds, even if sudo

b) you cannot delete "HSTS.plist ": if You type:

sudo rm -f ~/Library/Cookies/HSTS.plist

you get: Operation not permitted

c) even if you try:

ls -la ~/Library/Cookies/

you get: Operation not permitted

the same for

nano ~/Library/Cookies/HSTS.plist

(empty file..)

So you cannot definitely access it. (maybe SIP ?)

d) strangely you can delete if from Finder:

CMD Shift G "~/Library/Cookies/"

enter image description here

and you can delete with mouse:

e) more strange: You can move to Desktop using mouse, edit and place it back!

(a real nonsense, GUI is more powerful than sudo..)

that's due to the Full Disk Access permission as explained here https://apple.stackexchange.com/questions/227662/how-do-i-fully-flush-cached-redirects-from-safari#comment464819_288002 — Gobe, Oct 31 '22 at 18:14

score 1 · Answer 10 · answered Feb 04 '18 at 00:18

First make sure the server is not sending the Strict-Transport-Security header
You can do this with curl -I (-I just gets the headers)

curl -I http://my-http-domain.com

If the server is sending the Strict-Transport-Security header, removing it from your browser will have no effect, as next time you access the site, it will be set again.

Remove your site from Safari's Http Secure Transport Security database

Close Safari
Edit ~/Library/Cookies/HSTS.plist
Search for the entry for the site you want to access over http and remove it, and save the file.
- I prefer to edit rather than remove as there is no need to remove valid entries.
- I edit plist files using Xcode, but if it's not installed you can just use a text editor.
Restart your computer.
- Instead of restarting your computer, you can restart nsurlstoraged but that can be involved due to SIP so a computer restart may be simpler. See Grant's answer and Quanta's answer about restarting nsurlstoraged

score 1 · Answer 11 · answered Jul 17 '18 at 09:44

I made a script from Grand Heaslip's answer:

#!/bin/sh

osascript -e 'quit app "Safari"'
sleep 2
killall nsurlstoraged
sleep 2
rm -f ~/Library/Cookies/HSTS.plist
launchctl start /System/Library/LaunchAgents/com.apple.nsurlstoraged.plist

It gracefully ends safari, stops nsurlstoraged, removes the HSTS.plist, and starts nsurlstoraged again. This worked fine for me here on macOS 10.13.5

score 1 · Answer 12 · answered Oct 11 '18 at 01:49

I'm using Mojave (10.14). I tried the methods given so far to remove HSTS.plist. In addition, I needed to add Terminal to the System Preferences > Security & Privacy > Full Disk Access list to cure the symptom "Operation not permitted" when listing the contents of ~/Library/Cookies/.

But, removing the file and restarting the daemon didn't work. So I tried opening Safari again, went to Preferences, Privacy, Manage Website Data. Then I removed all "cache cookies, local storage" for the offending domain name. That solved my problem.

I can't say now if removing HSTS was required or not.

I've tried the same and rebooting twice, but only using Safari's UI worked for me as well. Thank you! — Bart Verkoeijen, Aug 23 '19 at 03:49

score 1 · Answer 13 · answered Jan 27 '23 at 19:27

Very simple as of Safari 16.2

Safari > Preferences > Privacy > Manage Website Data…
Search for the domain for which you want to reset HTTP redirections.
Click "Remove".
- Note: This removes "Cache, Cookies, Local Storage, and HSTS Policy" altogether for that particular domain. There is no method to delete only the cached HTTP redirections alone.

How I determined that this really works

On a web server under my control I used Wordpress with the Redirection plugin.

Wordpress > Admin Menu > Tools > Redirection > Log: Is empty.
I intentionally trigger a HTTP 301 redirection by entering URL-1 which gets redirected to URL-1-transformed.

Server side log shows URL-1 → URL-1-transformed with the timestamp of my request.
I invoke the same URL-1 again 2 more times.

Server side log does not show any corresponding log entries.

This means that the web browser has used its own HTTP redirection cache.
Cleared HTTP redirection cache for the domain as in the 1-2-3 instruction above.
Again requested URL-1 3 times. Always got redirected properly.

Server side log shows URL-1 → URL-1-transformed with the timestamp of the first request of my new request series.

That timestamp for the 1st new request proves that Safari had no more client-side HTTP redirection cache for the given domain and hence asked the server which then answered with a HTTP 301 redirection response.

The 2nd and 3rd request from that series then again must have happened via the client side HTTP cache as the server side log showed no corresponding redirection log entries.

Further background infos

After step #3 or #5 searching on disk for URL-1 or URL-1-transformed in the web browsers application data folders or caching folders yields nothing.

This cache seems to be solely in RAM.
And only get persisted rarely or possible not at all.

Possible reasons why the browser vendors don't persist this:

The redirection headers may have a short lifespan due their expiration-date being close.
Possibly also for performance reasons browsers keep this solely in RAM.
And overally due to the rather ephemer nature and potential undesired side effects the web browser developers possible do not consider it necessary/worthy to build methods to write/read this to/from persistent storage.

There are no known tools to inspect or manipulate cached HTTP redirections in:

Safari Web Inspector
Google Chrome DevTools — A similar thread on Google Chrome says: "It's unclear where Chrome stores the redirects."

How long to cache HTTP redirects also varies greatly across browsers.

Tested on Safari 16.2 (released 2022-12-13) on macOS 11.7.2 Big Sur (elder OS already). Appreciating confirmations of newer macOS versions.

Omi Harjani · Answer 14 · 2017-01-07T04:02:27.453

-1

Try this then, go to Step 1: Go to ~/Library folder, Step 2 :Delete Safari folder from ~/Library/Application Support, Step 3: Delete below folders from ~/Library/Caches, Step 4: then Delete ~/Library/Safari folder P.S:Keep safari closed during the above operations

edited Jan 07 '17 at 04:02

answered Jan 06 '17 at 09:41

Omi Harjani

101

1

Answers on Ask Different need to be more than just a link. It's okay to include a link, but please summarize or excerpt it in the answer. The idea is to make the answer stand alone. – nohillside Jan 06 '17 at 10:26

How Do I Fully Flush Cached Redirects From Safari?

14 Answers14

Older Safari

Remove your site from Safari's Http Secure Transport Security database

Very simple as of Safari 16.2

How I determined that this really works

Further background infos

Linked

Related