Script based way to load multiple delimited text files in QGIS

Question

I am generating heaps of model outputs from a scripted workflow. The most time consuming part is now loading the results 1 by 1 into QGIS for exploration. I found a way to load multiple files, but it doesn't include support for delimited text files (yet?). Then I have to set/load/copy the symbology, which is okay but still not ideal.

I'm wondering if there is a way to make "groups" from files in a folder. For (psuedo)example,

for d in ["fine","coarse"]:
    for f in os.listdir(d):
        if f.endswith('.head.head.csv'):
            <add delimited text file "f" to group "d">
            <apply head.qml style>
    <add group "d" to project.qgz>

To get a project with the following layers:

I have never dealt with QGIS files outside the GUI, so I'm not sure where to even start with the <> bits! I am digging through PyQGIS but can't find anything on delimited files.

Ben W · Accepted Answer · 2023-08-02T11:58:11.060

You can use the following script, or at least it should get you on your way. You will need to edit the first 5 variables after the import statements, as indicated in the script comments. These are the path to the folder containing your bulk CSV files, the names of the columns in the CSVs which contain the coordinates, the EPSG code of the correct CRS for your CSVs and the path to your style file. Of course, for this to work properly, the column names and CRS must be consistent for all CSV files. This scripts also assumes that the coordinates are in decimal format.

import os
from pathlib import Path

# *****Edit these 5 variables according to your file system and CSVs...*****#
# Path to your source directory...
source_folder = r'Path/To/Folder/Containing/CSVs'
# Name of CSV column containing Latitude values
lat_col = 'ycoord'
# Name of CSV column containing Longitude values
lon_col = 'xcoord'
# EPSG code of the correct CRS for your CSV coordinates
epsg_code = 'epsg:4326'
# Path to your .qml style file
style_file = r'Path/To/head.qml'
#***************************************************************************#

project = QgsProject.instance()

# Create the 2 groups in the layer tree if they do not already exist
fine_group = project.layerTreeRoot().findGroup('head fine')
if not fine_group:
    fine_group = project.layerTreeRoot().insertGroup(0, 'head fine')

coarse_group = project.layerTreeRoot().findGroup('head coarse')
if not coarse_group:
    coarse_group = project.layerTreeRoot().insertGroup(1, 'head coarse')

# Iterate over files in the source directory, create and load the CSV layers
for file in os.scandir(source_folder):
    file_path = os.path.join(source_folder, file.name)
    ext = os.path.splitext(file_path)[1]
    if ext == '.csv':
        file_stem = Path(file_path).stem
        uri = f'file:///{file_path}?type=csv&maxFields=10000&detectTypes=yes&xField={lon_col}&yField={lat_col}&crs={epsg_code}&spatialIndex=no&subsetIndex=no&watchFile=no'
        csv_lyr = QgsVectorLayer(uri, file_stem,'delimitedtext')
        if csv_lyr.isValid():
            # Load the style from the .qml file
            csv_lyr.loadNamedStyle(style_file)
            # Add layer to project but don't add to legend (we will add to groups next)
            project.addMapLayer(csv_lyr, False)
            # Add layer to correct group
            if 'fine' in file_stem:
                fine_group.addLayer(csv_lyr)
            else:
                coarse_group.addLayer(csv_lyr)

# Expand the groups in the layer tree view (optional)
view = iface.layerTreeView()
view.setExpanded(view.node2index(fine_group), True)
view.setExpanded(view.node2index(coarse_group), True)

Example result (These are just some dummy CSVs I created with names the same as yours):

Note: This is not a standalone script. This is simply run from the Python console and will add the layers and groups to the current project, inserting the layer tree nodes at the top of the layer tree.

By the way, in case you are wondering where this uri string comes from...

uri = f'file:///{file_path}?type=csv&maxFields=10000&detectTypes=yes&xField={lon_col}&yField={lat_col}&crs={epsg_code}&spatialIndex=no&subsetIndex=no&watchFile=no'

The basic idea is to load a CSV manually, select it and run:

print(iface.activeLayer().source())

This will print a valid source string for a single CSV. We can then take that string and modify it inside a for loop, by dynamically inserting some loop variables using Python's f-strings to construct valid source strings for every CSV in a directory.

Script based way to load multiple delimited text files in QGIS

1 Answers1