Searching directories and subdirectories to perform intersects and clips using ArcGIS Desktop?

Question

I'm trying to develop a tool using ArcGIS 10 that queries our company's data stores for any data that intersects a feature.

So that when we acquire a new property we can run this tool and it will find all of the data we have that intersects the new property, clip it, and save it in a new location.

Answer 1

A lot depends on how your data are stored. If your data store has grown organically and the data are jumbled in a load of directories and mix of shapefiles and geodatabases, you could use the Python os.walk function to list your directory structure and then subset your list to include only GDBs and SHPs. You would then need to recursively work through each dataset in the GDBs and each of the SHPs. This approach is a bit brute force but you could farm it out to a few CPU cores with a bit of threading and judicious use of the subprocess module.

If your data are neatly in a single database (whether that be a file-based geodatabase or SDE) your life should be a lot easier. You just then recursively walk through the database structure (using SQL and not the os.walk function obviously)

EDIT following Visceral's comments: Given that your directory tree includes mostly shapefiles you need to sift your files to only include the ones ending in '.shp'. Even if they ONLY contained shapefile, your are still only interested in the ones with the '.shp' extension. There's a few ways of doing this but here's an example:

def dirsearch(basedir):
    '''Gets all the shapefiles in the directory structure'''
    shpList = []
    for dirpath,dirnames,filenames in os.walk(basedir):
        shpList.extend([os.path.join(dirpath, fnm) for fnm in filenames if fnm.endswith('.shp')])
    return shpList

Answer 2

Here's what I used to do a quick count of GIS data on a server. Should be easy to modify for the intersect.

import arcpy
from arcpy import env
import os
env.workspace = "G:/data"

fcList = []
count = 0
for dirname, dirnames, filenames in os.walk('G:/data'):
    for subdirname in dirnames:
        print os.path.join(dirname, subdirname)
        env.workspace = os.path.join(dirname, subdirname)
        fcList = arcpy.ListFeatureClasses()
        for fc in fcList:
            print fc
            count = count + 1
            print count

Answer 3

I am not much of a python scripter but a client sent us about three hundred shapefiles in about 100 folders and I didn't have the time to extract them separately.

I created the model below using model builder and it works just fine.

Answer 4

This is more for ArcGIS 10.1 Service Pack 1 to current versions. Arcpy has a method called Walk which allows you to look through directories and sub-directory. The walk method is useful as you can specify the data types that you are looking for. The below is the example from the documentation. The example shows the Walk function being used to catalog polygon feature classes.

import arcpy
import os
workspace = "c:/data"
feature_classes = []
for dirpath, dirnames, filenames in arcpy.da.Walk(workspace,
                                              datatype="FeatureClass",
                                              type="Polygon"):
    for filename in filenames:
        feature_classes.append(os.path.join(dirpath, filename))

You could modify the code to loop through the catalogue, add a check to see if the features intersect and if true, clip the data.