Google Earth Engine: Get average of values by month and years using Python

Question

I am using python to download data from GEE.

I would like to get the average of the Land Surface Temperature of all the months over the years. This is what I am doing

def returnGEEAverages(satellite, subset, gemetry, name, sdate, edate):
    cover = ee.ImageCollection(satellite).filter(ee.Filter.date(str(sdate),
    str(edate))).select([subset])  
    pcts = cover.reduce(ee.Reducer.mean())
    task = ee.batch.Export.image.toDrive(
                                image=pcts,
                                folder=folderName,
                                region=geometry, 
                                scale=926.625433056, 
                                fileNamePrefix=name,
                                crs='epsg:4326')
    task.start()
    while task.active():
      t = task.id

I am doing two loops over the years and the months

 years = np.arange(2016, 2020, 1)
 for y in years:
  for months in range(1, 13):
    days = calendar.monthrange(y, months)
    sdate = date(y, months, 1)   # start date
    edate = date(y, months, days[1])   # end date
    folderName = urban_gdf_cities['UC_NM_MN'][k]
    geometry = returnGeometry(j)
    name = str(y)+str(months)
    lstDay = returnGEEPercentiles('MODIS/006/MYD11A1', 
    'LST_Day_1km', geometry, name+'LSTDay', sdate, edate)

I would like to know if there is a better way to do this. I found a similar solution but not for python.

Google earth engine:SST by month per year

score 4 · Answer 1 · answered May 07 '21 at 14:42

If you're trying to download a bunch of images cropped to your region, and that region is larger than about 1000x1000 pixels, then that method is roughly as good as anything else.

It's generally a bad plan to mix client-side objects (numpy/pandas) and server-side objects (ee.Image, etc), but if that's all the more computation you're doing it's probably not a big deal and it won't cause too many problems.

However, a canonical answer involving tasks and downloading images would look like this:

import ee
ee.Initialize()
def scaleAndMask(img):
  return (img.select("LST_Day_1km")
          .multiply(0.02)
          .updateMask(img.select("QC_Day").eq(0))
          .copyProperties(img, ["system:time_start"]))
def makeMonthlyComposite(date):
  date = ee.Date(date)
  return (collection
          .filterDate(date, date.advance(1, 'month'))
          .mean()
          .set("system:index", date.format("YYYY-MM")))
collection = ee.ImageCollection("MODIS/006/MOD11A2").map(scaleAndMask)
start = ee.Date('2016-01-01')
end = ee.Date('2016-02-01')
n_months = end.difference(start, 'month').subtract(1)
months = ee.List.sequence(0, n_months).map(lambda n : start.advance(n, 'month'))
result = ee.ImageCollection(months.map(makeMonthlyComposite))
ids = result.aggregate_array("system:index").getInfo()
for id in ids:
  image = result.filter(ee.Filter.eq("system:index", id)).first()
  task = ee.batch.Export.image.toDrive(...)
  task.start()

That said, if your output is smaller than 32MB, you can do things considerably faster by using getDownloadUrl and skipping the task queue altogether. This will also let you download the data directly to a numpy array, which seems like it might be where you're headed anyway:

import requests
import io
...
result = ee.ImageCollection(months.map(makeMonthlyComposite))
ids = result.aggregate_array("system:index").getInfo()
for id in ids:
  image = result.filter(ee.Filter.eq("system:index", id)).first()
  url = image.getDownloadUrl({'region': region, 'scale': 1000, 'format': "NPY"})
  response = requests.get(url)
  data = numpy.load(io.BytesIO(response.content))
  print(data)

Sounds really interesting, thank you. At this point would it be possible to convert the array data as a geopandas dataframe with crs='epsg:4326'? — emax, May 11 '21 at 09:28
Not familiar with geopandas; no idea how to convert in-memory data blocks into something it understands. You can download GeoTIFFs this way though by just changing the format to TIFF. At worst, you could write that to disk and read it with their file readers. But be aware that the size limit is real; if they're too large, you could end up with corrupt files. — Noel Gorelick, May 11 '21 at 09:53

Google Earth Engine: Get average of values by month and years using Python

1 Answers1