Calls to SAGA routines in QGIS not working cross platform

Question

The ability to write and run cross platform scripts for QGIS using python is OUTSTANDING! However, when I try to run the SAGA:MERGESHAPESLAYERS algorithm from a python script it works fine in windows but returns no output shape when I run it in ubuntu. I'm running QGIS 2.4 on ubuntu.

I'm confident that I DO have SAGA 2.1 on the ubuntu system because I can successfully run the convex hull algorithm.

To illustrate the issue I have written the following scripts. The first of which creates a few of shape files in the temp directory then the second attempts to apply the aforementioned SAGA algorithms. You should be able to easilty cut and paste the scripts into the QGIS python console in order to illustrate the issue.

First script which creates a few shape files in the temp directory:

#BUILD EXAMPLE SHAPE FILES TO ILLUSTRATE THE ISSUE
from qgis import core
from qgis.utils import iface
import os
import tempfile
import sys
epsg=4326
my_crs =core.QgsCoordinateReferenceSystem(epsg,core.QgsCoordinateReferenceSystem.EpsgCrsId)
iface.mapCanvas().mapRenderer().setDestinationCrs(my_crs)
vlayer01=QgsVectorLayer("Polygon?crs=epsg:"+str(epsg)+"&field=id:integer&field=name:string(20)&index=yes","polygon_01","memory")
bbl_dp = vlayer01.dataProvider()
vlayer01.startEditing()
feature_01_01 = QgsFeature()
feature_01_01.setGeometry(QgsGeometry.fromPolygon(\
[[QgsPoint(0,100),QgsPoint(0,110), QgsPoint(10,110),QgsPoint(10,100)]]))
feature_01_02 = QgsFeature()
feature_01_02.setGeometry(QgsGeometry.fromPolygon(\
[[QgsPoint(-10,100),QgsPoint(-10,110), QgsPoint(-20,110),QgsPoint(-20,100)]]))
(res, outFeats) = vlayer01.dataProvider().addFeatures([feature_01_01,feature_01_02])
vlayer01.commitChanges()
QgsMapLayerRegistry.instance().addMapLayer(vlayer01)
vlayer01_path=os.path.join(tempfile.gettempdir(),"vlayer_01.shp")
write_error = \
QgsVectorFileWriter.writeAsVectorFormat(vlayer01, \
vlayer01_path, "system", \
QgsCoordinateReferenceSystem(epsg), \
"ESRI Shapefile")
vlayer02=QgsVectorLayer("Polygon?crs=epsg:"+str(epsg)+"&field=id:integer&field=name:string(20)&index=yes","polygon_02","memory")
bbl_dp = vlayer02.dataProvider()
vlayer02.startEditing()
feature_02_01 = QgsFeature()
feature_02_01.setGeometry(QgsGeometry.fromPolygon(\
[[QgsPoint(20,110),QgsPoint(20,120), QgsPoint(30,120),QgsPoint(30,110)]]))
(res, outFeats) = vlayer02.dataProvider().addFeatures([feature_02_01])
vlayer02.commitChanges()
QgsMapLayerRegistry.instance().addMapLayer(vlayer02)
vlayer02_path=os.path.join(tempfile.gettempdir(),"vlayer_02.shp")
write_error = \
QgsVectorFileWriter.writeAsVectorFormat(vlayer02, \
vlayer02_path, "system", \
QgsCoordinateReferenceSystem(epsg), \
"ESRI Shapefile")
vlayer03=QgsVectorLayer("Polygon?crs=epsg:"+str(epsg)+"&field=id:integer&field=name:string(20)&index=yes","polygon_03","memory")
bbl_dp = vlayer03.dataProvider()
vlayer03.startEditing()
feature_03_01 = QgsFeature()
feature_03_01.setGeometry(QgsGeometry.fromPolygon(\
[[QgsPoint(3,103),QgsPoint(3,113), QgsPoint(13,113),QgsPoint(13,103)]]))
(res, outFeats) = vlayer03.dataProvider().addFeatures([feature_03_01])
vlayer03.commitChanges()
QgsMapLayerRegistry.instance().addMapLayer(vlayer03)
vlayer03_path=os.path.join(tempfile.gettempdir(),"vlayer_03.shp")
write_error = \
QgsVectorFileWriter.writeAsVectorFormat(vlayer03, \
vlayer03_path, "system", \
QgsCoordinateReferenceSystem(epsg), \
"ESRI Shapefile")
qgis.utils.iface.setActiveLayer(vlayer01)
qgis.utils.iface.zoomToActiveLayer()

Second script which addresses the issue

###SAGA PROCESSING ALGORITHM ISSUE STARTS HERE###
import processing
alg_list=[]
alg_list.append("qgis:mergevectorlayers")
alg_list.append("saga:mergeshapeslayers")
alg_list.append("saga:convexhull")
for alg in alg_list:
    if alg=="saga:convexhull":
        SHAPES=vlayer01_path
        POLYPOINTS=0
        HULLS=os.path.join(tempfile.gettempdir(),"output_saga_convexhull.shp")
        BOXES=""
        processing_result=processing.runalg(alg,SHAPES,POLYPOINTS,HULLS,BOXES)
        print alg+" was "+str(processing_result)
        resultlayer = QgsVectorLayer(HULLS, "saga:convexhull", "ogr")
        QgsMapLayerRegistry.instance().addMapLayer(resultlayer)
        qgis.utils.iface.setActiveLayer(resultlayer)
        qgis.utils.iface.zoomToActiveLayer()
    if alg=="saga:mergeshapeslayers":
        MAIN=vlayer01_path
        LAYERS=vlayer02_path+";"+vlayer03_path
        OUT=os.path.join(tempfile.gettempdir(),"output_saga_mergeshapeslayers.shp")
        processing_result=processing.runalg(alg,MAIN,LAYERS,OUT)
        print alg+" was "+str(processing_result)
        resultlayer = QgsVectorLayer(OUT, "saga:mergeshapeslayers", "ogr")
        QgsMapLayerRegistry.instance().addMapLayer(resultlayer)
        qgis.utils.iface.setActiveLayer(resultlayer)
        qgis.utils.iface.zoomToActiveLayer()
    if alg=="qgis:mergevectorlayers":
        LAYER1=vlayer01_path
        LAYER2=vlayer02_path
        SAVENAME=os.path.join(tempfile.gettempdir(),"output_qgis_mergevectorlayers.shp")
        processing_result=processing.runalg(alg,LAYER1,LAYER2,SAVENAME)
        print alg+" was "+str(processing_result)
        resultlayer = QgsVectorLayer(SAVENAME, "qgis:mergevectorlayers", "ogr")
        QgsMapLayerRegistry.instance().addMapLayer(resultlayer)
        qgis.utils.iface.setActiveLayer(resultlayer)
        qgis.utils.iface.zoomToActiveLayer()

The merge algorithm works only in windows and the convex hull algorithm works in both windows and ubuntu. I'd really like the SAGA:MERGESHAPESLAYERS algorithm to work cross platform as intended.

Is that script intendend to run inside or outside (standalone) of QGIS GUI? As far as I know, it was not possible to call QGIS processing tools from outside QGIS (see question here: http://gis.stackexchange.com/questions/71820/access-sextante-processing-in-standalone-qgis-app). I would be very happy to learn that this has changed. — Martin, Oct 20 '14 at 06:45
Oops I used the word "standalone" without realizing it was jargon. Edited. — Mr Purple, Oct 20 '14 at 07:23
Maybe an issue of backslash vs forward slash: http://gis.stackexchange.com/questions/34631/how-to-link-to-external-files-with-relative-path-in-qgis-actions — AndreJ, Oct 20 '14 at 07:49
I don't thinks it's a path 'slash' issue. I use os.path.join to build my paths. In any case, it wouldn't explain why the convex-hulls algorithm works but the merge doesn't. Well that's my code. I can't speak for the QGIS Ubuntu implementation of each the SAGA algorithms. — Mr Purple, Oct 20 '14 at 07:55
Does the merge algorithm work in Ubuntu when started manually from QGIS? — AndreJ, Oct 20 '14 at 08:00
No. I get the "Oooops! The following output layers could not be open" error and the "SAGA seems to be correctly installed and configured" comment. The log file closes before I can get a look at it. From what I can see before it closes nothing looks odd. — Mr Purple, Oct 20 '14 at 08:08
Please check if the "Use SAGA 2.0.8 syntax" option is checked out in the processing options. It's a common issue with running SAGA 2.1 commands from QGIS. — Gabor Farkas, Oct 20 '14 at 08:11
Sorry, I found that issue before and can confirm that I have unchecked "processing/options and configuration/providers/SAGA/Use SAGA 2.0.8 syntax" — Mr Purple, Oct 20 '14 at 08:14
In Processing -> Options, General you can activate to keep the dialogue open. — AndreJ, Oct 20 '14 at 08:15
Can you check what caused the problem exactly from the log? When I tried to run r.drain from QGIS some days ago, it ended up in the same error, and the log said that python couldn't import a core plugin for raster handling. — Gabor Farkas, Oct 20 '14 at 08:20
Kept the log open (thanks AndreJ) and read the whole log. No errors that I could see. It ended with Converting outputs Loading resulting layers Algorithm Merge shapes layers finished — Mr Purple, Oct 20 '14 at 08:35
NB: I select layers when run from within QGIS and files when run from script. I cant see how to run multiple layers from the script as I cant concatenate layer handles with the ";" string. Even if I use only two layers in the script I get no output result layer. — Mr Purple, Oct 20 '14 at 08:40
After further inspection of the log I see "Unknown option 'MAIN'". When I run SAGA from a terminal I get a similar problem. Its as though SAGA 2.1.2 in ubuntu doesn't like the command as generated by QGIS for the shapes_tools "Merge Shapes Layers" process. — Mr Purple, Oct 22 '14 at 05:31

Mr Purple · Accepted Answer · 2014-10-23T05:48:41.373

I may have found why the call to SAGA:MERGESHAPESLAYERS is not working in ubuntu from QGis

Regardless of my 2.0.8 syntax setting QGis generates the following line of shell script

shapes_tools "Merge Shapes Layers" -MAIN "/tmp/vlayer_01.shp" -LAYERS "/tmp/vlayer_01.shp;/tmp/vlayer_02.shp" -OUT "/tmp/output.shp"

When run it generates the following error:

Unknown option 'MAIN'

and further down the error log I note:

Usage: saga_cmd shapes_tools 0

Which is the "Create New Layer" tool and has no 'MAIN" input. When I enter the following line directly into my shell:

saga_cmd shapes_tools

I note that in my version of SAGA (2.1.2) I see no "Merge Shapes Layers" tool. Thus the request for a "Merge Shapes Tool" appears to be defaulting to tool no. 0 The "Create New Shape Layer" tool.

There does appear to be a merge tool, but it's called "Merge Layers" not "Merge Shapes Layers"

So my solution (in ubuntu - which uses SAGA 2.1.2) is going to have to be to first determine if I have SAGA version 2.1.2 or higher then manually prepare and call the "Merge Layers" tool.

Here's my work around (working in my ubuntu).The shell commands will probably need tweaking to work in windows. If I find myself in windows I'll fix that up too.

#REPLACE YOUR PREVIOUSLY WORKING
#processing_result=processing.runalg("saga:mergeshapeslayers",MAIN,LAYERS,OUT)

#WITH NOW WORKING (IN UBUNTU)
import subprocess
from subprocess import Popen, PIPE
from distutils.version import StrictVersion
p = Popen(["saga_cmd", '-v'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
saga_version=output[14:].rstrip()
if StrictVersion(saga_version) >= StrictVersion('2.1.2'):
    INPUT=MAIN+";"+LAYERS
    MERGED=OUT
    p = Popen(["saga_cmd", "shapes_tools","2","-INPUT",INPUT,"-MERGED",MERGED], stdin=PIPE, stdout=PIPE, stderr=PIPE)
    output, err = p.communicate(b"input data that is passed to subprocess' stdin")
    rc = p.returncode
else:
    processing_result=processing.runalg("saga:mergeshapeslayers",MAIN,LAYERS,OUT)

Honestly, I hope I'm wrong but if not, hopefully QGis can be patched to correctly generate scripts for this powerful algorithm ASAP.

To get the issue fixed, you better report it at http://hub.qgis.org/projects/quantum-gis/issues. Developers are reading over there, but only seldom here. — AndreJ, Oct 23 '14 at 04:55
Thanks AndreJ. I'm not all signed up for that but you are welcome to report it and link to here if you want. — Mr Purple, Oct 23 '14 at 05:51
I will try to reproduce the error by myself, and then report it. Which Ubuntu version and which repository did you use to install QGIS and SAGA? — AndreJ, Oct 23 '14 at 06:03
software repo http://qgis.org/debian ubuntu 14.04 trusty thar, Qgis 2.4, SAGA 2.1.2 as per http://gis.stackexchange.com/questions/72957/installing-qgis-2-0-and-saga-2-1-on-ubuntu-13-04/73335#73335 — Mr Purple, Oct 23 '14 at 09:15
I got a reply from the processing developer: http://lists.osgeo.org/pipermail/qgis-developer/2014-October/035253.html. It seems that current Windows and Linux versions of SAGA are a bit apart, and rapidly changing from version to version. — AndreJ, Oct 24 '14 at 11:08
Thanks to the team at QGIS this error (and my subsequent workaround) are only valid for 2.4. QGIS 2.6 is working fine. — Mr Purple, Nov 02 '14 at 09:00

Calls to SAGA routines in QGIS not working cross platform

1 Answers1