EMR serverless, new operator

[gumartinm]

Description

A new EMR serverless has been announced and it is already available, see:

• https://aws.amazon.com/blogs/big-data/announcing-amazon-emr-serverless-preview-run-big-data-applications-without-managing-servers/
• https://aws.amazon.com/emr/serverless/

Having an operator for creating applications and submitting jobs to EMR serverless would be awesome.

Use case/motivation

New operator for working with EMR serverless.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[yeshbash]

Looks like the feature is not Generally Available and I couldn't find the docs to the APIs. I'd be happy to pitch in with this operator if there are more details about it. Also, any guidance from the community is appreciated.

[ferruzzi]

That's correct, this is in preview and there is no announced date for general release yet. I'll also be happy to help once the docs are available.

[dacort]

@yeshbash @ferruzzi If you haven't already, please sign up for the preview here: https://pages.awscloud.com/EMR-Serverless-Preview.html

We've started to let folks in and you will receive info on how to access the preview when you're allowlisted.

Docs are here and I've also created an emr-serverless-samples repository that may be useful.

I'm on the EMR team and would be happy to help implement or review any PRs. 😁

[treyyi]

Hi @dacort,

I am also interested in EMR Serverless features for airflow such as operators, sensors, or testings.
Once EMR serverless is released officially, I will try to provide some suggestions, and I am willing to help you even if it wouldn't be my PR!
(FYI, currently I am in data engineer position for operating EMR clusters and airflow server.)

[dacort]

I just published some information about how to use (a custom, while in preview) boto3 to access the EMR Serverless API.

https://github.com/aws-samples/emr-serverless-samples/tree/main/examples/python-api

[dacort]

Working on moto PR as well so we can get ahead of tests. :) getmoto/moto#5071

[dacort]

Hi folks - we're working internally at AWS on this. If folks here are currently in the preview and interested in testing this, please reach out to me. I'm dacort on Twitter/LinkedIn/and the Airflow Slack.

[treyyi]

@dacort I am in the preview now, so just sent you a message on Twtitter!