Efficient downsampling with GDAL?

Question

I have a 350go GeoTIFF file with DEM data for the entire world. (a mix of SRTM, EUDEM, etc. ...)

This TIFF file is tiled with a block size of 256*256.

I would like to downsample this file to a lower resolution (original file resolution is 1295829, 464402)

I tried with this command:

gdal_translate -co NUM_THREADS=ALL_CPUS -r nearest -co BIGTIFF=YES -co 
COMPRESS=LZW -outsize 0.1% 0.1% -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 -co 
TILED=YES world.tif downsized_world.tif

The output file resolution should be close to 1295 * 464

The problem is that it seems like it will take forever to complete (progress bar stuck at 0 for few hours) do you have any idea on how I could resample this file faster?

Resampling the file with an even lower resolution does not seem to make it go faster when for me it should be, isn't it? (Fewer pixels to read = faster resample?)

That should be rather simple operation. I would rip off parameters NUM_THREADS, BIGTIFF, and COMPRESS but probably the speed remains the same. Do you have overviews in your big image? — user30184, Sep 12 '18 at 13:33
They would certainly help even what you want to do should really work even without. External overviews can be created with gdaladdo -ro -r average --config COMPRESS_OVERVIEW LZW world.tif if you have GDAL 2.3. For earlier versions add also a list of overview levels 2 4 8 16 32 64 128 256 https://www.gdal.org/gdaladdo.html — user30184, Sep 12 '18 at 15:05

pLumo · Accepted Answer · 2018-09-12T14:17:29.087

4

Try gdalwarp, it can utilize multithreading and might be more efficiant than gdal_translate for this kind of tasks:

gdalwarp -r average -ts 1295 464 \ 
    -wm 2048 -multi -wo NUM_THREADS=ALL_CPUS \
    -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 -co TILED=YES \
    -co NUM_THREADS=ALL_CPUS -co COMPRESS=LZW \
    world.tif downsized_world.tif

Although average will be slower, nearest Interpolation is not suitable to be used with DEM data.

Change the used memory to fit your system, it's this option: -wm 2048.

edited Sep 12 '18 at 14:17

answered Sep 12 '18 at 14:05

pLumo

6,489
20
41

Thanks i'll try it right now ! what is the fastest interpolation if nearest does not work for DEM data ? – Duke Sep 12 '18 at 14:24
Nearest will work and is fastest, but it will create "wrong" data. It might not be a problem for you though. Average is a simple operation so it's pretty fast too, something like cubic or lanczos might be a bit nicer but is considerably slower. You could also consider med. See documentation. – pLumo Sep 12 '18 at 14:32
Nearest is taking the value of the nearest pixel right ? if that's the case I think it's enough for me. How much time do you think,I should expect this command to run for ? (minutes ? hours ? days ?) – Duke Sep 12 '18 at 14:49
Depends a lot on your system ... maybe up to few hours, but much less than a day. I'm currently running a gdalmerge on 15TB of data which takes roughly two days. That system has a decent octacore. – pLumo Sep 12 '18 at 14:51
1

Ok thanks a lot! I'll let this command run and I'll let you know of the result in a few hours ;) – Duke Sep 12 '18 at 14:53
1

It took a few hours to complete on my machine, but yes it's way faster than with gdal_translate, thanks again ! – Duke Sep 13 '18 at 09:03

Efficient downsampling with GDAL?

1 Answers1