9

It seems like processing a large raster dataset in QGIS can cause a crash, for example in QGIS failing to complete clip or intersect and crash dump in qgis. Is it possible to use GDAL parallel processing for a large raster dataset? Is there any example for this?

das-g
  • 1,445
  • 2
  • 11
  • 33
HMadadi
  • 1,046
  • 11
  • 24
  • 1
    This is a useful thread on the topic – dmci Jan 28 '16 at 21:39
  • Is your question about vector or raster dataset ? You are linking to a thread on vectors, then you mention GDAL and raster. Please be more specific in your question. – radouxju Feb 27 '18 at 13:48
  • https://stackoverflow.com/questions/27805554/gdal-function-to-split-large-image-into-specific-chunks – klewis Feb 27 '18 at 16:14
  • I would suggest a old blog posts: https://artlembo.com/2016/07/22/qgis-gpu-processing-update/ and related plugin: https://plugins.qgis.org/plugins/CUDARaster/ btw qgis classes are not completly thread safe (better qgis3 thatn qgis2) => so can be not predictable the parallel result on the same instance... multiple instance works – Luigi Pirelli Feb 27 '18 at 16:27
  • Whenever I need to process large amounts of data I run gdal from the command line. Check http://www.gdal.org for info, examples and a couple tutorials to get you comfortable using it from the command line. – sig_renato Jan 28 '16 at 21:46
  • That's right but when you want use in plugin, what do you do? – HMadadi Jan 29 '16 at 11:28
  • I have a bit of a plugin fobia… :( – sig_renato Jan 29 '16 at 14:35
  • this does not really answer the question whether gdal can be used in parallel... – joaoal Nov 13 '17 at 10:30

1 Answers1

2

GDAL can use multiple threads for some processing. For example gdal2tiles https://gdal.org/programs/gdal2tiles.html can create base tiles in multiple processes. Gdalwarp https://gdal.org/programs/gdalwarp.html can also use multiple processes.

When it comes to clipping, I am not sure if multithreading could yield big benefits. Clipping is rather simple process, to find correct source pixels and to write them into a target file. Multithreading would probably require that it is possible to write data into several places of the target file. Perhaps it could be possible with very well known format like uncompressed tiff. But I think that there is now something in QGIS that is using unnecessary much resources and multithreading is not the right place to start with. For example, is your target file tiled?

user30184
  • 65,331
  • 4
  • 65
  • 118