NDVI computation in Lambda
|4 months ago|
|.gitattributes||5 months ago|
|.gitignore||5 months ago|
|.pre-commit-config.yaml||5 months ago|
|.python-version||4 months ago|
|README.md||4 months ago|
|doeberitzer_heide.geojson||5 months ago|
|doga.jpg||5 months ago|
|doga.py||4 months ago|
|poetry.lock||4 months ago|
|pyproject.toml||4 months ago|
|test_doga.py||4 months ago|
DOGA is a simple map analysis tool using up42 technologies. Doga means "nature" in Turkish but there's "ğ" instead of "g".
┌────────────┐ ┌─────────────────────────────────────────┐ │ │ │ │ │ DOGA_CLI ├──────────► DOGA_LAMBDA │ │ │ GET │ │ └──┬─────────┘ POST │ ┌───────────────┐ ┌──────────────┐ │ │ │ │ ├────► │ │ │ │ │ API GATEWAY │ │ API LAMBDA │ │ │ GET │ │ ◄────┤ │ │ │ POST │ └───────────────┘ └──────────────┘ │ │ │ │ │ └─────────────────────────────────────────┘ ┌──▼─────────────────┐ │ │ │ ELEMENT84 │ │ EARTH SEARCH API │ │ │ └────────────────────┘
- Python 3.7.9 (to use the same version with Geolambda)
- AWS (Lambda, S3, API Gateway, Cloudwatch, IAM, etc.)
poetry install --no-dev poetry run doga --help
One more step.. When you try to use doga with a geojson file, then you will get a warning like this one:
poetry run doga -i doeberitzer_heide.geojson -c sentinel-s2-l2a-cogs --start 2020-01-01 --end 2020-12-31 Please review your configuration. You can see more information on the project page. Aborted!
So we need to create a
.env file in the project folder and define some variables:
In case you will need to know what the API url and secret key, you will have this environment values in the next step Server Side.
If you are working at UP42, please contact me directly to get the
.env file for accessing the pre-installed AWS Lambda or you can install it to your AWS account.
The data that we want to process is too big so we need a cloud-native solution. Basically, we have two components: AWS Lambda is for running our functions in the cloud, and AWS API Gateway is for accessing the functions from our CLI app. Let's start with an easy one.
AWS Lambda Installation
We have a helper command to make an archive to upload and deploy to AWS Lambda. It will prepare an archive named
poetry run export
If you don't have an experience in AWS Lambda, please take a look at the external sources section.
The second step, we need some layers to use GDAL
And last, please define a variable named
DOGA_LAMBDA_API_SECRET to accept CLI requests. That can be a random value, that's not a problem. But please do not forget to define the same variable on your
AWS API Gateway Configuration
To make some requests from the CLI to the cloud functions, we need an API Gateway. Here you can find an introduction how you can configure the API gateway for your Lambda functions.
After you completed the instructions, you will have an API url. Please define a variable named
DOGA_LAMBDA_API_URL using this url, including the endpoint path.
You can install all the dependencies using poetry:
For your code changes, please do not forget to check the linting and run unit tests:
poetry run pre-commit run poetry run pytest
Better, you can add pre-commit to git commit hooks with this command:
poetry run pre-commit install
What could we do more?
- Static Typing support using Mypy & Pyright
- Using a coverage tool to get more information about the scope of the unit tests
- Bitbucket Pipelines integration
- Docker containerization as an alternative installation
Do we really need AWS Lambda?
Indeed no. We could install all the dependencies with
poetry but in case we must also consider how we will compile or integrate gdal, numpy or similar scientific libraries. Of course, we could use WSL2 or we could prepare a Docker image but all of these options have some system requirements. So I preferred a cloud-native solution and I wanted to keep client-side requirements at a minimum.
Do we really need a CLI app?
Yes or no, it depends. I realized that some scientists are using Anaconda & Conda package manager instead of pure Python and I see that there are many similar command-line applications. Maybe that would be cool to write also a web-based app or a simple REST API that calculates NDVIs. Why not?