Add Kerberos support

[garthcn]

It'd be great if Airflow could add Kerberos support to hooks (e.g. HDFS, etc). The snakebite lib Airflow depends on just added Kerberos support in this PR: spotify/snakebite#153

[mistercrunch]

I think that an impersonate flag in BaseOperator could be a decent generic way to help around security and attribution of "who is running what", even if Airflow is triggering the job.

The way impersonate would work is that the job on the worker would sudo su {task.owner} prior to running it. This assumes unix based roles and security to be deployed on workers. Of course many systems (take mysql for instance) don't support unix-type authentication.

Impersonate could be forced to True for given operators using the undocumented policy feature described in settings.py

[txominpelu]

Hello @garthcn and @mistercrunch , I would be really interested in having this feature in airflow. Right now the lack of support for kerberos is the main thing that is blocking the adoption of airflow in my project.

I am willing to help in the implementation. Has anybody already started working on it?

[mistercrunch]

Can you describe which part(s) of Airflow you are thinking about integrating with Kerberos? Specific operators? The whole platform!?

[txominpelu]

I'd like to modify some operators (e.g Hive) so that they can be scheduled and run with the right user in a hadoop cluster in secure mode .

[pharoson]

#229 is a similar issue. It includes the current workaround I'm using to run workflows as a different user.

[mistercrunch]

@txominpelu , I'm not sure if you read the impersonation section in here:
http://www.cloudera.com/content/cloudera/en/documentation/cdh4/v4-3-0/CDH4-Security-Guide/cdh4sg_topic_9_1.html

You could have a model where the user on their dev box need to provide their kerberos creds, and in production the server has superuser creds and impersonates the owner of the task.

[bolkedebruin]

I did the work in snakebite. I am also interested in having Kerberos in. And although I need to wrap my mind around how airflow works and how it could be integrated I am
Willing to spend some time on it. I would think of something like:

Airflow has its own principal and keytab. This allows it to renew tickets on behalf of users, so it allows users to use their own credentials for "submitting". I'm not sure yet though how the submitting part would work. Or in other words how to make sure scripts executed under the users credentials are secured
And someone else cannot impersonate the user. Any ideas?

[bolkedebruin]

Btw having only one super user is not enough in a multi tenant environment. Segregation of duties is required then. Therefore separate users are required

[txominpelu]

I was thinking on doing something similar to what hue does in the default setup for cloudera. The hue user is a proxy user that can impersonate others when launching oozie workflows, hive queries, spark jobs, etc. That means that the hue user can do anything (not ideal), but hue users can only schedule and run things by asking hue to impersonate them.

[mistercrunch]

@txominpelu , this should work. The key is users develop using their own credentials locally, and when the code runs on the worker it runs as user airflow that can impersonate the owner of the task.

There needs to be safeguards around source control, and the policy can be inforced with the "policy function" on the worker.

[bolkedebruin]

@txominpelu actually the hue user is not capable of doing everything except that it is allowed to impersonate other users when the right credential is supplied for that user. Ideally airflow, in a Kerberos context, should never execute under its own principal.

[bolkedebruin]

The question for me is where to start? Basically airflow should not allow unauthenticated users to 1) connect to the webservice (ie present authentication screen) 2) start/schedule jobs of other users. Also how are users going to submit jobs? In the end they cannot copy their code to the server I presume or maybe they can? And what to do with the user administration? Does it need synchronization or externalization?

We can start small of course. First Kerberize the webservice only allow authenticated users, leaving user admin for what it is. Then move on to the command line and jobs and maybe force the user to the one submitting the job?

[mistercrunch]

One of the first step is for the hook to allow other types of authentication and make the impersonation work.
https://github.com/airbnb/airflow/blob/master/airflow/hooks/hive_hooks.py#L329

[bolkedebruin]

Impersonation is the one of the most complex steps. Ie it requires a service principal, a ticket renewer and verification of the user's credentials.