2

I've recently tried using Shapely to dissolve and intersect large sets of 100'000+ geometries, some of them quite complex. This would take me at least one hour, at which point I would usually shut the process down.

I've been recommended PostGIS a few times as a super fast alternative, and I am wondering if it would indeed be faster? From my understanding both PostGIS and Shapely use GEOS, so would there really be such a speed difference between the two?

Jochen Schwarze
  • 14,605
  • 7
  • 49
  • 117
Chouroud
  • 97
  • 4
  • 4
    You should try and find out. Speed will depend on you precise scenario. But the benefits of PostGIS will be enhanced indexing (I don't know if you are using spatial indexes with Shapely), better data access (PostgreSQL will handle it efficiently). – HeikkiVesanto Nov 29 '18 at 14:08
  • 1
    @HeikkiVesanto Thanks for the answer. I agree that it's best to try out, but there is a cost to learning and setting up PostgreSQL and PostGIS. If it ends up saving me only a few minutes per dataset, I'm not really sure it's worth experimenting with it. – Chouroud Nov 29 '18 at 14:45
  • 1
    There is a cost to learning and setting up everything. I am a Pythonista and Postgres/Postgis lover, so no deliberate bias here, but I would argue that for larger datasets and more complex queries, the Postgres/Postgis environment is a better environment, plus you get all the RDBMS benefits thrown in. Learning of alternative ways of solving a problem is always a good idea. – John Powell Nov 30 '18 at 09:12
  • 1
    anecdotal: I recently had to intersect a set of 100 million polygons with a set of 33 million. My first pass at a Shapely solution was looking like it would have taken months to execute, and my PostGIS/Django solution takes about a day to run. – Teddy Ward Nov 30 '18 at 21:09
  • @TeddyWard great to hear. I'm very keen on making the leap to PostGIS. Any idea what makes it faster? Is it the chunk-based data access that makes all the difference? I'm coming from a programming background so I'm curious what's going on in the backend! – Chouroud Nov 30 '18 at 21:28
  • Foir what it's worth I've done some preliminary tests and it looks like implementing something in Shapely along these lines helps a lot. Let's see if that reduces the speed enough. – Chouroud Nov 30 '18 at 21:30
  • There is no magik. If your data is spatially indexed, it's faster. Is Shapely able to index your polygons? – Pierre Racine Dec 03 '18 at 14:07
  • @PierreRacine Yes, I shapely seems to be able to spatially index my geometry. – Chouroud Dec 04 '18 at 15:07

0 Answers0