ogr2ogr get geojson from map server misses features and creates duplicates

Question

I'm querying this map service using ogr2ogr to create a local copy of it's features: http://geoportal.menlhk.go.id/arcgis/rest/services/KLHK/Deforestasi_2006_2009/MapServer/0

Here's my ogr2ogr code, using where clause objectid > 0 to select everything:

ogr2ogr 2006_2009.geojson -f GEOJson "http://geoportal.menlhk.go.id/arcgis/rest/services/KLHK/Deforestasi_2006_2009/MapServer/0/query?where=objectid>0&outfields=*&f=json" OGRGeoJSON -gt 1000 -s_srs EPSG:3857 -t_srs EPSG:4326

My output (2006_2009.geojson) has 8724 features, which matches the record count here: http://geoportal.menlhk.go.id/arcgis/rest/services/KLHK/Deforestasi_2006_2009/MapServer/0/query?where=1%3D1&returnCountOnly=true&f=json

However, when I compare the shapefile in Arc with the shapefile in ArcOnline (coming directly from the source map server) I found the shapefile I created was missing features.

The reason I was seeing matching feature counts is because the ogr2ogr output has duplicate records:

ogrinfo 2006_2009.geojson OGRGeoJSON -dialect sqlite -sql "SELECT objectid, count(objectid) FROM OGRGeoJSON GROUP BY objectid HAVING (COUNT(objectid) > 1)"

The above query identifies 147 instances of duplicate objectids, suggesting that we're missing 147 records of real data given that the record counts match. Looking at the output dataset, I can see that there's nothing for objectids between 852 and 892. I can use ogr2ogr to extract these features successfully:

ogr2ogr -f GeoJSON missing_features.geojson "http://geoportal.menlhk.go.id/arcgis/rest/services/KLHK/Deforestasi_2006_2009/MapServer/0/query?where=objectid%3E852%20AND%20objectid%3C892&outFields=*&f=json" OGRGeoJSON -gt 1000 -s_srs EPSG:3857 -t_srs EPSG:4326

But for some reason these are not present in the original output, where the where clause selected all features with objectids > 0.

In sum, two issues:

1. Why doesn't ogr2ogr export objectids 852 - 892 (and others) given a where clause of objectid > 0?
2. More troubling-- why does it duplicate features to make the record counts equal?

Answer 1

There may be a bug in the GDAL GeoJSON driver http://www.gdal.org/drv_geojson.html affecting how the paging works but I am not quite sure. It seems that you have tried to set paging to 1000 features with parameter -gt 1000. However, -gt affects writing, not reading. The paging size of ESRI REST service is set, as documented in another way:

The page size can be explicitly set with the resultRecordCount parameter (but is subject to a server limit). If it is not set, OGR will set it to the maximum value allowed by the server.

I believed that this modified query gives you a correct result. It also prints the debug info and the exact queries which are sent. You can copy the queries and send them with your browser and continue debugging that way.

ogr2ogr 2006_2009_2.geojson -f GEOJson "http://geoportal.menlhk.go.id/arcgis/rest/services/KLHK/Deforestasi_2006_2009/MapServer/0/query?where=objectid>0&outfields=*&f=json&resultRecordCount=1000" OGRGeoJSON -s_srs EPSG:3857 -t_srs EPSG:4326 --debug on

What may be a GDAL bug is that it actually finds that the server has a feature limit and makes queries accordingly. And it does so that after the first queries are sent with parameters

resultRecordCount=1000&resultOffset=1000
resultRecordCount=1000&resultOffset=2000
resultRecordCount=1000&resultOffset=3000
resultRecordCount=1000&resultOffset=4000
resultRecordCount=1000&resultOffset=5000
resultRecordCount=1000&resultOffset=6000
resultRecordCount=1000&resultOffset=7000
resultRecordCount=1000&resultOffset=8000

However, the first query is sent without record count and it seems that ESRI server is sending different first 1000 records when the query is set without resultRecordCount parameter and when query is sent with resultRecordCount=1000 and without resultOffset. It may be that ESRI server is sorting the records differently and here is the possible GDAL bug: it could be safer to discard the first batch of data and make a new request with resultRecordCount=1000&resultOffset=0 and force the ESRI server to sort the records.

Answer 2

Try this for controlling your request

query?where=1+<+1000&outfields=*&f=pjson" OGRGeoJSON