1

I have a shape file with 1,900,589 rows associated to it. I attempt to make a copy of it by exporting it, and the total row count drops significantly. I just did a generic scroll through the data, by doing a ctrl-shft-end, and it looks like all the records are legit.

-RAM usage on my PC never gets above 50% during the export. (16gb ram on my machine)

In ArcMap 10.7, after the export, the row count drops down to 1,600,260

-The method I'm using to export, is right clicking on the file in the table of contents, Data, Export Data..., Exporting All Features, changing the file name.

-The original file size is 2.07GB

-Post export file size is 2.06GB (Losing 300k rows seems like it should have had a more significant effect on this file size?)

In QGIS 3.24.3, after the export, the row count drops down to 1,299,930

-Weird thing about QGIS, when I open the table, at the top, it says Features total 1,900,589, Filtered 1,299,930. I have nothing filtered...

I also read about a possible bug where you had to change the "Show All Features button" to "Show Features Visible On Map". (Which still showed 1,299,930 records)

I'm going to try splitting this file down into a few smaller chunks, and see if it behaves any differently, but that is not going to stop the pain of me not know why it's doing this.

Vince
  • 20,017
  • 15
  • 45
  • 64
vbj4
  • 11
  • 2
  • 1
    Shape file have a (theoretical : https://gis.stackexchange.com/questions/312739/why-are-shapefiles-limited-to-2gb-in-size) size limit of 2gb, you seem to be right on it so your problem may come from that. Did you try to export to a more suitable format for large file ? – J.R Apr 19 '23 at 15:36
  • 1
    That, at least makes sense. I can live with too beaucoup. Thank you! I suspect this is probably the issue. I reloaded the original CSV into QGIS, and it actually has 2,501,248 rows. Suspecting I lost some when converting it from csv to shape in ArcMap. QGIS seems to be handling it fine now that I'm avoiding the .shp.

    Thanks again

     – vbj4 Apr 19 '23 at 16:00

0 Answers0