goedev
/
doga
Archived
1
0
Fork 0
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
This repo is archived. You can view files and clone it, but cannot push or open issues/pull-requests.
Gökmen Görgen 6ab9d57e89 Merged in feature-3 (pull request #5) 12 months ago
.gitattributes add argument parsing support for doga cli. #4 12 months ago
.gitignore ignore archives. #6 12 months ago
.pre-commit-config.yaml Merged in feature-1 (pull request #1) 12 months ago
.python-version decrease python version to 3.7.9. #5 12 months ago
README.md decrease python version to 3.7.9. #5 12 months ago
doeberitzer_heide.geojson make a new command for exporting project to AWS lambda. #5 12 months ago
doga.jpg add argument parsing support for doga cli. #4 12 months ago
doga.py first implementation of ndvi computation. #3 12 months ago
poetry.lock make unit tests asyncio compatible. #6 12 months ago
pyproject.toml make unit tests asyncio compatible. #6 12 months ago
test_doga.py make unit tests asyncio compatible. #6 12 months ago

README.md

DOGA

DOGA is a simple map analysis tool using up42 technologies. Doga means "nature" in Turkish but there's "ğ" instead of "g".

Technical Skills

┌────────────┐          ┌─────────────────────────────────────────┐
│            │          │                                         │
│  DOGA_CLI  ├──────────►  DOGA_LAMBDA                            │
│            │    GET   │                                         │
└──┬─────────┘    POST  │  ┌───────────────┐    ┌──────────────┐  │
   │                    │  │               ├────►              │  │
   │                    │  │  API GATEWAY  │    │  API LAMBDA  │  │
   │ GET                │  │               ◄────┤              │  │
   │ POST               │  └───────────────┘    └──────────────┘  │
   │                    │                                         │
   │                    └─────────────────────────────────────────┘
┌──▼─────────────────┐
│                    │
│  ELEMENT84         │
│  EARTH SEARCH API  │
│                    │
└────────────────────┘
  • Python 3.7.9 (to use the same version with Geolambda)
  • AWS (Lambda, S3, API Gateway, Cloudwatch, IAM, etc.)

Installation

Install Python and Poetry first. Then:

poetry install --no-dev
poetry run doga --help

One more step.. When you try to use doga with a geojson file, then you will get a warning like this one:

poetry run doga -i doeberitzer_heide.geojson -c sentinel-s2-l2a-cogs --start 2020-01-01 --end 2020-12-31

Please review your configuration. You can see more information on the project page.
Aborted!

So we need to create a .env file in the project folder and define some variables:

DOGA_LAMBDA_API_URL="https://xxx.amazonaws.com/doga"
DOGA_LAMBDA_API_SECRET="4S3cr3tK3y!"

In case you will need to know what the API url and secret key, you will have this environment values in the next step Server Side.

If you are working at UP42, please contact me directly to get the .env file for accessing the pre-installed AWS Lambda or you can install it to your AWS account.

Server Side

The data that we want to process is too big so we need a cloud-native solution. Basically, we have two components: AWS Lambda is for running our functions in the cloud, and AWS API Gateway is for accessing the functions from our CLI app. Let's start with an easy one.

AWS Lambda Installation

We have a helper command to make an archive to upload and deploy to AWS Lambda. It will prepare an archive named doga_lambda.zip:

poetry run export

If you don't have an experience in AWS Lambda, please take a look at the external sources section.

The second step, we need some layers to use GDAL

And last, please define a variable named DOGA_LAMBDA_API_SECRET to accept CLI requests. That can be a random value, that's not a problem. But please do not forget to define the same variable on your .env file.

AWS API Gateway Configuration

To make some requests from the CLI to the cloud functions, we need an API Gateway. Here you can find an introduction how you can configure the API gateway for your Lambda functions.

After you completed the instructions, you will have an API url. Please define a variable named DOGA_LAMBDA_API_URL using this url, including the endpoint path.

Contribution

You can install all the dependencies using poetry:

poetry install

For your code changes, please do not forget to check the linting and run unit tests:

poetry run pre-commit run
poetry run pytest

Better, you can add pre-commit to git commit hooks with this command:

poetry run pre-commit install

What could we do more?

  • Static Typing support using Mypy & Pyright
  • Using a coverage tool to get more information about the scope of the unit tests
  • Bitbucket Pipelines integration
  • Docker containerization as an alternative installation

Discussions

Do we really need AWS Lambda?

Indeed no. We could install all the dependencies with poetry but in case we must also consider how we will compile or integrate gdal, numpy or similar scientific libraries. Of course, we could use WSL2 or we could prepare a Docker image but all of these options have some system requirements. So I preferred a cloud-native solution and I wanted to keep client-side requirements at a minimum.

Do we really need a CLI app?

Yes or no, it depends. I realized that some scientists are using Anaconda & Conda package manager instead of pure Python and I see that there are many similar command-line applications. Maybe that would be cool to write also a web-based app or a simple REST API that calculates NDVIs. Why not?

External sources