Catching a system exit with GDAL Python

Question

I am writing a small program that runs through a ton of different shape files, tab files and geodatabases and gets their extent (about 10000- 100000 files).

I am opening these using the GDAL python library. I am using Python 2.7 and GDAL 2.2.3 on a windows 7 platform.

Unfortunately, some of these layers are completely corrupted, which causes GDAL to crash with a system error code.

ERROR:root:Could not read layer g:\31\23922\Tech\SWMM\Heatherton\Data\Heatherton_NS_draped.TAB

Process finished with exit code -1073741819 (0xC0000005)

My code looks like:

import ogr, logging, osr
path = r'g:\31\23922\Tech\SWMM\Heatherton\Data\Heatherton_NS_draped.TAB'
def check_path(path):
    try:
        logging.debug(path)
        ds = ogr.Open(path)
        if ds is None:
            return None
        for i in range(ds.GetLayerCount()):
            print ds.GetLayerByIndex(i).GetExtent()
    except:
        import traceback
        traceback.print_exc()
check_path(path)
print 'amIhere?'

I am fine with skipping this file. I just want to reach the print 'amIhere?' part. Or best case scenario: reach the except block.

But the fact that it uses an exit code, makes the entire thing crash. Is there a way to capture this?

I have tried catching SystemExit, BaseException....

Answer 1

I have managed to do a workaround by using subprocesses. Not ideal, but it works.

import ogr, logging, osr
from multiprocessing import Process, Queue

path = r"C:\pgtest\test\1.shp"
def check_path(path, newQueue):
    try:
        logging.debug(path)
        ds = ogr.Open(path)
        if ds is None:
            return None
        for i in range(ds.GetLayerCount()):
            newQueue.put(ds.GetLayerByIndex(i).GetExtent())
    except:
        import traceback
        traceback.print_exc()


if __name__ == '__main__':
    newQueue = Queue()
    p = Process(target=check_path, args=(path,newQueue,))
    p.start()
    p.join()
    if not newQueue.empty():
        print newQueue.get()
    print 'amIhere?'

The solution does slow the process down a lot! I am open to any suggestions.

Answer 2

Lennert, there's no exception to catch in your case. The gdal and ogr modules don't raise Python exceptions by default. I believe that if you add import gdal; gdal.UseExceptions() at the top of your script, you'll get a normal Python exception that you catch and handle in a single Python process (no multiprocessing needed). The situation with GDAL and exceptions is explained here: https://pcjericks.github.io/py-gdalogr-cookbook/gdal_general.html#enable-python-exceptions.