Use QGIS `processing.run(...., alg_params)` from OSGEO4W shell in QGIS Dev 3.26

Question

I've created a simple tool using the code from save model to automate processing. It runs as expected in the QGIS python console but does not work from osgeo4w shell.

I have tried the process in https://towardsdatascience.com/how-to-use-qgis-spatial-algorithms-with-python-scripts-4bf980e39898 and https://www.qgistutorials.com/en/docs/running_qgis_jobs.html

I have also tried Spyder in Conda but I keep getting ModuleNotFoundError: No module named 'qgis'

Even looking at similar questions here, I see it's been an issue for different versions and so they don't work either.

--- UPDATE

It ended up being some install error and uninstalling and re-installing QGIS worked. To run the python to allow for import qgis to work you have to open the bat file - C:\OSGeo4W\bin\python-qgis.bat and then you can import qgis however processing.run is not part of it so I still need to know how to run algs as below.

---

I need to run algs like

from qgis.core import *
# Points to path
    alg_params = {
                'CLOSE_PATH': False,
                'GROUP_EXPRESSION': '',
                'INPUT': outputs['CreatePointsLayerFromTable']['OUTPUT'],
                'NATURAL_SORT': False,
                'ORDER_EXPRESSION': '',
                'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
            }
    outputs['PointsToPath'] = processing.run('native:pointstopath', alg_params)
    print ('3_PointsToPath')

Is there a guide like the examples above?

I want to automate the running of this py file to process data weekly. So alternatively is there a way to launch qgis with this py file to process and close?

The whole code that I want to run in qgis_processing is below.

import os, glob, re
from qgis.core import *
in_dir=r'Z:/Mowing_Automated_Reporting/Input'
out_dir=r'Z:/Mowing_Automated_Reporting/Working/Draft_Output'
os.chdir(in_dir)
def Clean_Mower_Data(in_file,out_file, max_speed, max_length):
    # Extract by expression
    outputs = {}
    max_speed_alg=''"Speed KmH"  < '+ str(max_speed)+''' 
    print (max_speed_alg)
    alg_params = {
                'EXPRESSION': '"Speed KmH"  < '+ str(max_speed) ,
                'INPUT': in_file,
                'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT #'Z:/My Drive/Mangoesmapping/Spatial Projects/2022/DSC/015_DSC_Mowing_Automated_Reporting/working/1.tab',
                }
outputs['ExtractByExpression']=processing.run('native:extractbyexpression', alg_params)
print ('1_ExtractByExpression')

# Create points layer from table
alg_params = {
            'INPUT': outputs['ExtractByExpression']['OUTPUT'],
            'MFIELD': '',
            'TARGET_CRS': QgsCoordinateReferenceSystem('EPSG:4326'),
            'XFIELD': 'Longitude',
            'YFIELD': 'Latitude',
            'ZFIELD': '',
            'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
        }
outputs['CreatePointsLayerFromTable'] = processing.run('native:createpointslayerfromtable', alg_params)
print ('2_CreatePointsLayerFromTable')

# Points to path
alg_params = {
            'CLOSE_PATH': False,
            'GROUP_EXPRESSION': '',
            'INPUT': outputs['CreatePointsLayerFromTable']['OUTPUT'],
            'NATURAL_SORT': False,
            'ORDER_EXPRESSION': '',
            'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
        }
outputs['PointsToPath'] = processing.run('native:pointstopath', alg_params)
print ('3_PointsToPath')

# Explode lines
alg_params = {
            'INPUT': outputs['PointsToPath']['OUTPUT'],
            'OUTPUT': QgsProcessing.TEMPORARY_OUTPUT
        }
outputs['ExplodeLines'] = processing.run('native:explodelines', alg_params)
print ('4_ExplodeLines')

# Extract by expression
alg_params = {
            'EXPRESSION': '$length &lt;'+str(max_length),               
            'INPUT': outputs['ExplodeLines']['OUTPUT'],
            'OUTPUT': out_file
        }
processing.run('native:extractbyexpression', alg_params)
print ('5_Extracted')


# Fix geometries
alg_params = {
'INPUT': outputs['ExtractByExpression2']['OUTPUT'],
'OUTPUT': out_file
}
processing.run('native:fixgeometries', alg_params)
print ('6_Fixed')
filelist=glob.glob('*.csv')
for file in filelist:
#Process with speed/length limits
fn=os.path.splitext(file)[0]
max_speed=30
max_length =150
fn=s=re.findall(&quot;^DeviceDataExport_(.*)_\d*-\d*-\d*.*&quot;,fn)
fn=re.sub('[^A-Za-z0-9]+_', '', fn[0])+'__s'+str(max_speed)+'_l'+str(max_length)
in_file=os.path.join(in_dir,file)
out_file=out_dir+&quot;//&quot;+fn+&quot;.tab&quot;
Clean_Mower_Data(in_file,out_file, max_speed, max_length)
print ('processed: %s --to--&gt; %s' % (file, fn))

#Process without speed/length limits
fn=os.path.splitext(file)[0]
max_speed=999
max_length =99999
fn=re.findall(&quot;^DeviceDataExport\s(.*)\s\d*-\d*-\d*.*&quot;,fn)
fn=re.sub('[^A-Za-z0-9]+', '', fn[0])
in_file=os.path.join(in_dir,file)
out_file=out_dir+&quot;//&quot;+fn+&quot;.tab&quot;
Clean_Mower_Data(in_file,out_file, max_speed, max_length)
print ('processed: %s --to--&gt; %s' % (file, fn))

Answer 1

To get this to work I had to do the following.

0 - Make sure there is nothing wrong with your install by first running C:\OSGeo4W\bin\python-qgis.bat and then try import qgis. If this gives errors the following will not work.

1 - Create a bat file with the code from C:\OSGeo4W\bin\python-qgis.bat and add the name of the python file at the end.

2 - Add the code from https://gis.stackexchange.com/a/279937/2891 to the python file. Note the settings for the QgsApplication.setPrefixPath and sys.path.append.

There a few things with using memory and a proper structure for commands to work. Will post details once I work it all out.

3 - Add a scheduled task as per https://www.qgistutorials.com/en/docs/running_qgis_jobs.html

So my code for a windows system is

BAT file

@echo off
call "C:\OSGeo4W\bin\o4w_env.bat"
@echo off
path %OSGEO4W_ROOT%\apps\qgis\bin;%PATH%
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT:\=/%/apps/qgis
set GDAL_FILENAME_IS_UTF8=YES
rem Set VSI cache to be used as buffer, see #6448
set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000
set QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins;%OSGEO4W_ROOT%\apps\qt5\plugins
set PYTHONPATH=%OSGEO4W_ROOT%\apps\qgis\python;%PYTHONPATH%
python3 test_qgis.py
pause

Python file

import sys
from qgis.core import (
     QgsApplication, 
     QgsProcessingFeedback, 
     QgsVectorLayer
)
See https://gis.stackexchange.com/a/155852/4972 for details about the prefix
QgsApplication.setPrefixPath('C:\OSGeo4W\apps\qgis', True)
qgs = QgsApplication([], False)
qgs.initQgis()
Append the path where processing plugin can be found
sys.path.append('C:\OSGeo4W\apps\qgis\python\plugins')
import processing
from processing.core.Processing import Processing
Processing.initialize()
layer1 = QgsVectorLayer('/path/to/geodata/lines_1.shp', 'layer 1', 'ogr')
layer2 = QgsVectorLayer('/path/to/geodata/lines_2.shp', 'layer 2', 'ogr')
You can see what parameters are needed by the algorithm
using: processing.algorithmHelp("qgis:union")
params = { 
    'INPUT' : layer1,
    'OVERLAY' : layer2, 
    'OUTPUT' : '/path/to/output_layer.gpkg|layername=output'
}
feedback = QgsProcessingFeedback()
See https://gis.stackexchange.com/a/276979/4972 for a list of algorithms
res = processing.run('qgis:union', params, feedback=feedback)
res['OUTPUT'] # Access your output layer