2

So we're running into an issues where TempDB's log is currently locked up. After checking log_reuse_wait_desc, we see that it is locked from an active transaction. Now the weird part is that when we run DBCC OPENTRAN to get information on the query causing the lock, the command returns "No active open transactions". We've killed every spid connecting to TempDB and we're at the point where we'll be cycling the instance to resolve the issue. Does anyone understand what's happening here or if there's a better way to deal with this?

8bit
  • 388
  • 1
  • 11
  • 2
    What do you see in sys.dm_tran_active_transactions? Just because DBCC OPENTRAN returns 0 rows doesn't mean no transactions exist. Try this: USE msdb; BEGIN TRANSACTION; SELECT * INTO #foo FROM dbo.sysjobs; Now try DBCC OPENTRAN... – Aaron Bertrand Dec 02 '15 at 23:37
  • 1
    Also, what exactly do you mean the log is "locked up"? Are you trying to shrink it? Why? – Aaron Bertrand Dec 02 '15 at 23:38
  • This will remove all cache plans,etc from buffer pool... So use it with caution.Before restarting try DBCC FREEPROCCACHE -- clean cache DBCC DROPCLEANBUFFERS -- clean buffers DBCC FREESYSTEMCACHE ('ALL') -- clean system cache DBCC FREESESSIONCACHE -- clean session cache. Also, check sessions using sys.dm_exec* dmv. – Kin Shah Dec 02 '15 at 23:40
  • John, do you mean you have other sessions waiting on that one tempdb transaction? – Nicolas de Fontenay Dec 02 '15 at 23:45
  • @AaronBertrand Locked up was probably a poor term to use, rather, it's not re-using space. The TempDB files are all set sizes with no auto-growth and users are getting errors when running queries that use temp tables. We're not trying to Shrink it, I'm just trying to figure out why we cant find whatever spid is locking it up. I ran the query you mentioned above and I'm still getting nothing in DBCC OPENTRAN :/ – 8bit Dec 02 '15 at 23:46
  • @Kin - We'll give that a try before we cycle tonight. – 8bit Dec 02 '15 at 23:46
  • Your tempdb log file is a fixed size with auto-growth disabled? Why? And that was my point - DBCC OPENTRAN does not report all transactions. – Aaron Bertrand Dec 02 '15 at 23:50
  • @ndefontenay - It's just causing horrible slowdowns. Things that take seconds on our dev servers are taking hours in production, but there are no resource spikes and the cache looks fine when looking at Spotlight – 8bit Dec 02 '15 at 23:50
  • This http://www.sqlservercentral.com/scripts/tempdb/72007/ will give you the amount of tempdb space used by active sessions. If nothing turns up, the space is not used by active sessions, which makes it trickier to troubleshoot. – spaghettidba Dec 02 '15 at 23:50
  • @AaronBertrand - Got it, I see what you were getting at. As for the auto-growth disable & fixed sizing, it's a decision that pre-dates me that i'm trying to get over-turned. Right now, it's what I have to deal with :/ – 8bit Dec 02 '15 at 23:51
  • Well, if you uncheck that checkbox, this problem goes away, at least buying you some time, so... – Aaron Bertrand Dec 02 '15 at 23:52
  • Can you query sys.dm_tran_active_snapshot_database_transactions and see if you have any orphaned transaction -- or - you can download sp_whoisactive and run it with @show_sleeping_spid = 1(run with @help =1 to show other help parameters … since I am not sure if it is @show_sleeping_spid .. I can check later). – Kin Shah Dec 03 '15 at 04:21

2 Answers2

2

So we figured it out:

We rebooted the server and when it came back up, we were still having issues. We started re-considering whether it was just an issue with TempDB and turned to maybe trying to catch the issues with extended events. While looking into it, we stumbled across an extended event that nobody knew about called "Testing." We disabled that XE and, go figure, everything started working fine again. I'm not sure WHY this locked up everything in TempDB and we're looking into understanding the issue, but for now - the issue is resolved.

8bit
  • 388
  • 1
  • 11
  • we stumbled across an extended event that nobody knew about called "Testing." - Testing should not be on PROD :-) – Kin Shah Dec 03 '15 at 22:54
  • Oh, things I know @Kin. If only there were a way to track who created it... – 8bit Dec 03 '15 at 22:56
0

Can you see what's running in tempdb?

;WITH task_space_usage AS (
    -- SUM alloc/delloc pages
    SELECT session_id,
           request_id,
           SUM(internal_objects_alloc_page_count) AS alloc_pages,
           SUM(internal_objects_dealloc_page_count) AS dealloc_pages
    FROM sys.dm_db_task_space_usage WITH (NOLOCK)
    WHERE session_id <> @@SPID
    GROUP BY session_id, request_id
)
SELECT TSU.session_id,
       TSU.alloc_pages * 1.0 / 128 AS [internal object MB space],
       TSU.dealloc_pages * 1.0 / 128 AS [internal object dealloc MB space],
       EST.text,
       -- Extract statement from sql text
       ISNULL(
           NULLIF(
               SUBSTRING(
                 EST.text, 
                 ERQ.statement_start_offset / 2, 
                 CASE WHEN ERQ.statement_end_offset < ERQ.statement_start_offset 
                  THEN 0 
                 ELSE( ERQ.statement_end_offset - ERQ.statement_start_offset ) / 2 END
               ), ''
           ), EST.text
       ) AS [statement text],
       EQP.query_plan
FROM task_space_usage AS TSU
INNER JOIN sys.dm_exec_requests ERQ WITH (NOLOCK)
    ON  TSU.session_id = ERQ.session_id
    AND TSU.request_id = ERQ.request_id
OUTER APPLY sys.dm_exec_sql_text(ERQ.sql_handle) AS EST
OUTER APPLY sys.dm_exec_query_plan(ERQ.plan_handle) AS EQP
WHERE EST.text IS NOT NULL OR EQP.query_plan IS NOT NULL
ORDER BY 3 DESC;
Nicolas de Fontenay
  • 1,873
  • 2
  • 18
  • 36
  • 2
    I think I know this query :-) – spaghettidba Dec 02 '15 at 23:50
  • 2
    This will only return active requests, not those that are idle/sleeping. This could be a BEGIN TRANSACTION; SELECT ...; that "finished" last Tuesday, and it won't show up in sys.dm_exec_requests... – Aaron Bertrand Dec 02 '15 at 23:50
  • Like mentioned above, all this returns are some of the active transactions we can track - none of which seem to be causing the lock (Just a few spids that have kicked off in the past hour or so) – 8bit Dec 02 '15 at 23:58
  • 1
    The query is referenced from @AaronBertrand 's answer – Kin Shah Dec 03 '15 at 04:15
  • Yes I had found it here. Is that yours spaghettidba? I have seen a few good things from you peppered across dba.stackexchange. I should read all comments first. @AaronBertrand I had to use it a lot when we struggled with ASP State database causing pain in tempdb. – Nicolas de Fontenay Dec 03 '15 at 18:40
  • What @spaghettidba was gently trying to tell you is you should have stated where you got the SQL. Notice how Aaron provided an attribution in the original post. – Erik Dec 04 '15 at 03:08
  • No problem, I'm fine with anybody using my code, as long as they find it useful. – spaghettidba Dec 04 '15 at 09:19