Converting large CSV to line-delimited GeoJSON?

Question

What is a good way to convert a large (100MB) CSV file to GeoJSON on the command line, preferably without GDAL? I tried Mapbox's csv2geojson but it ran out of memory.

Steve Bennett · Answer 1 · 2018-01-31T21:58:28.010

Since I'm comfortable with Node, I came up with this stream-oriented script. It's easy to transform values in the process.

#!/usr/bin/env node
const fs = require('fs');
const turf = require('@turf/turf');

function rowToFeature(record, callback) {

    // Adjust these field names according to your file
    if (!(record.Latitude && record.Longitude)) {
        return;
    }
    let feature = turf.point([ +record.Longitude, +record.Latitude ], {
        myfield: record['My Input Field'],
        joinedfield: record.field1 + record.field2
        // ... etc
    });
    callback(null, JSON.stringify(feature) + '\n');
}

fs.createReadStream('infile.csv')
    .pipe(require('csv-parse')({ columns: true }))
    .pipe(require('stream-transform')(rowToFeature))
    .pipe(fs.createWriteStream('outfile'));

Before running:

npm init
npm install csv-parse stream-transform

+1 As a 'Node guy', I like this solution. There is a useful module in the NPM repository called dbGeo which takes data usually from a DB query but it cold equally be a parsed CSV file and does a similar thing to this. — MappaGnosis, Jan 30 '18 at 09:03

score 3 · Answer 2 · answered Jan 30 '18 at 00:03

This could be a job for CSVKit, my new favourite tool:

https://source.opennews.org/articles/eleven-awesome-things-you-can-do-csvkit/

5. Turn a CSV with latitude and longitude columns into GeoJSON

Is your data geographic, like in our previous example? Then don’t stop with JSON—go one step further and make it GeoJSON!

**csvjson --lat latitude --lon longitude --key slug --crs EPSG:4269 --indent 4 geo.csv > geo.jso**n

[geo.json]

{
    "type": "FeatureCollection",
    "bbox": [
        -95.31571447849274,
        32.299076986939205,
        -95.28174,
        32.35066
    ],
    "features": [
        {
            "type": "Feature",
            "id": "dcl",
            "geometry": {
                "type": "Point",
                "coordinates": [
                    -95.30181,
                    32.35066
                ]
            },
            "properties": {
                "place": "Downtown Coffee Lounge"
            }
        },
        {
            "type": "Feature",
            "id": "tyler-museum",
            "geometry": {
                "type": "Point",
                "coordinates": [
                    -95.28174,
...

Reference: csvjson

Thank you, that works. It's very slow though. On my 100MB file, it takes nearly 4 minutes whereas my script (which saves few properties) takes 35 seconds. — Steve Bennett, Jan 30 '18 at 05:12
In my script, I'm stripping out properties that I don't want in the output. Using this tool keeps them. So it's maybe not a totally fair comparison. — Steve Bennett, Jan 31 '18 at 22:00

score 1 · Answer 3 · edited Feb 13 '18 at 06:24

1

Another node solution that would require less coding than Steve Bennett's excellent solution would be to use the NPM Module called csv2geojson.

There is also another modules called csv3geojson. This version purports to be faster than csv2geojson and supports streaming so maybe better in this particular use-case. I can't vouch for it though as I have not used it, but certainly worth a look.

Also see Converting CSV file to GeoJSON while preserving data types? which is a similar Q&A.

edited Feb 13 '18 at 06:24

PolyGeo

65,136
29
109
338

answered Jan 30 '18 at 09:07

MappaGnosis

33,857
2
66
129

Thanks. As I mentioned in the Q, csv2geojson ran out of memory for me. I tried to increase the maximum heap size with an environment variable but I'm not sure it had any effect. A streaming version would be likely to work. – Steve Bennett Jan 31 '18 at 21:48

Converting large CSV to line-delimited GeoJSON?

3 Answers3