7

I have point shapefiles in 50,000 different folders (each shapefile file name is unique) - about 33,400 shapefiles (not all folders will have a point shapefile). I like to open the point shapefile in each of the folder into QGIS Desktop 2.2.0 map frame and then hopefully run a merge on them to get a single point shapefile layer. I tried drag-dropping from Windows Explorer to QGIS, and QGIS froze initially and continue to add the point shapefiles for several hours and didn't seem to stop. I found the Load Them All plugin (http://goo.gl/HbJhhS) that seem to specify the feature type and also specify a string filter, this plugin also didn't seem to work...

I was thinking that had I planned well, I could've automated and copied the point shapefiles into another single folder as soon as they were created and I could 've dragged-dropped into mapframe from this single folder. I don't know if this would've had worked. My analysis run took 18 days to run, so I don't want to re-run just to do that. I tried copy-pasting with Windows Explorer and it hung, may be because it was 100,000+ files...

Would you know any trick to do this?

PolyGeo
  • 65,136
  • 29
  • 109
  • 338
Arun Govind
  • 375
  • 2
  • 9

3 Answers3

9

I advise merging your shapefiles into a single folder (using a program or search/copy/paste or a small script such as the one Mark C recommends in the comments). Then use this rather elegant bit of code from gis-programming.com:

as above, use: ogr2ogr merge.shp file1.shp to create a shapefile merge containing the data of file1

then:

 for %f in (*.shp) do ( 

 ogr2ogr -update -append merge.shp %f  -f “esri
 shapefile” -nln merge 

 )
Community
  • 1
Barrett
  • 3,069
  • 1
  • 17
  • 31
  • That is really very elegant! – WhiteboxDev Aug 13 '14 at 22:12
  • It would be nice to bring the shapes into a single folder, but if he uses a GUI, there could be a problem when the files exceed 65536 or something like that. There are ~50,000 shape files to deal with in this case. It would be easier to recursively search each folder and test for its geometry type instead. – SaultDon Aug 13 '14 at 22:32
  • See HERE for a Dos Batch File to copy all the files into one common directory – Mark Cupitt Aug 14 '14 at 01:35
  • This is a good answer, @ArunGovind. And if it's germane, I additionally recommend reviewing the answers in this question if it's worthwhile to add a new field to preserve/distinguish which points originally belonged to each of your merged shapefiles. – elrobis Aug 14 '14 at 15:36
2

Here is a simple solution for those facing a similar problem in R:

library(sf)
library(magrittr)

# find all shape files
  all_shape_addresses <- list.files(path="R:/root_folder", pattern = ".shp", full.names =T, recursive = TRUE)

# read all files into a list
  files_list <- lapply(all_shape_addresses, st_read)

# Merge all files into a single object
  single <- Reduce(st_union, x)

# save single shape file
  st_write(single , dsn = "single.shp", layer = "single.shp", driver = "ESRI Shapefile")

I'm not sure how the speed would compare to the solutions presented in the other answers. The performance of the solution in R might be restricted due to memory constraints in case there are too many and too large shape files.

rafa.pereira
  • 1,267
  • 17
  • 27
1

I got this to work. But thank you to everyone who answered and their awesome ideas! I'll explore each of them.

I used Global Mapper which has the option to search directories and its sub-directories for selected file types (File > Open all files in a directory tree). It took around 12 hours to load the shapefiles from 50,000 sub-folders and then I selected point or polygon layer and separately exported them as a new vector layer. The export took 15-30 minutes for each layer.

All points and polygons

Selected points and polygons zoomed in

Arun Govind
  • 375
  • 2
  • 9