10

I have a very large ESRI shapefile (>8000 polygons, .shp file is >32MB), which I want to read into R on Windows 7.

I am familiar with rgdal and readOGR(). However, with this size of shapefile, reading the entire shapefile into memory is a very slow process. Furthermore, I am only interested in a few clustered polygons (<100) of the 8,000 that are contained in the shapefile. The process for which I need to do this is highly repetitive, i.e. the shapefile needs to be loaded many times, so it is currently very slow.

Hence, I am wondering if it is possible to only read in a part of the shapefile in order to accelerate the process. It seems unpractical to load such a massive amount of data and then discard most of it.

rcs
  • 3,894
  • 1
  • 27
  • 30
bmz
  • 101
  • 4
  • What platform are you using? If you have access to ArcGIS, I suppose you could script something in ArcPy that makes the subset selection and then executes an R process from the python script. This is definitely possible... Run a Google search for executing R code from python. – GeoJohn Dec 02 '16 at 15:25
  • 3
    Maybe try export into new shapefile that part of your data which is important to you, and then work on it in R. – ami Dec 02 '16 at 16:00
  • it will probably work with sf. But save it to R workspace format with saveRDS. There are ways to store generically in SQLite or similar, but I doubt that is required – mdsumner Dec 03 '16 at 09:41

3 Answers3

8

The only thing that comes to mind is to leverage the new "sfr" library and its associated simple feature class for the subsetting. The sfr library is currently available on GitHub and here is a tutorial to get you started.

I do not believe that you can stream data based on a subset query however, this would at lease speed up read times considerably and is supposedly the future of R spatial classes.

Another option would be to store your spatial data in a database (eg., PostGIS) and query from the database. This can be an extremely efficient way to deal with large data.

Jeffrey Evans
  • 31,750
  • 2
  • 47
  • 94
  • 2
    The sf package is available on CRAN since version 0.2 (2016-10-26): https://cran.r-project.org/web/packages/sf/index.html – rcs Dec 02 '16 at 21:22
1

I think that using a GIS such as QGIS and selecting out the features you need would be a good approach.
If you need to merge back the results you could create one file with the 7900 polygons that you don't want to process, then process the 100 polygons, and rejoin after processing.

MichaelStoner
  • 790
  • 5
  • 13
0

You can make a subset of a shapefile using the org2ogr command, as exemplified here:

ogr2ogr selecting features by attributes

You'll need to figure out a condition for selecting the features you want though.

Community
  • 1
Spacedman
  • 63,755
  • 5
  • 81
  • 115