spark-submit job is not exiting until I hit ctrl+C if I use Check class

[bruce32118]

Is your feature request related to a problem? Please describe.

spark-submit job is not exiting until I hit ctrl+C if I use Check function

Describe the solution you'd like

Check function would start a self._spark_session.sparkContext._gateway server. If my code is completed, the server is still hang on that causes the spark job can't stop normally.

Adding a close function to class Check

def close(self):
self._spark_session.sparkContext._gateway.close()
return

[MOHACGCG]

I am experiencing the same issue. once the deequ run is complete and the results are gathered, pydeequ seems to keep something running in background. closing spark session doesn't help.

EDIT:
spark.sparkContext._gateway.close()
spark.stop()

does the job.

[gucciwang]

Thanks for the suggestion! We will add this feature into the next release :)

[datadominik]

@gucciwang @jaoanan1126 , any update here? Running hasSize checks from within AWS Glue will not work without adding spark.sparkContext._gateway.close() manually for me. Would appreciate a cleaner solution here. Not sure though if the spark.stop() is required.

[AbdelrahmanAli]

For whoever reading the issue here, I think it's worth mentioning this doc https://pydeequ.readthedocs.io/en/latest/README.html#wrapping-up

[ketankvishwakarma]

spark.sparkContext._gateway.close()
spark.stop()

Not working for me

[apwrk]

Hi,

Is there a permanent solution to this problem other than the hacky solution given here?

[github-actions]

This is a known issue caused by the Py4J callback server that gets started in PythonCallback.__init__ (in scala_utils.py) whenever a ScalaFunction1 is created — which happens every time you use assertion lambdas in Check methods like hasSize, hasMin, etc. The callback server keeps the JVM-to-Python communication channel alive, preventing the process from exiting.

The documented workaround is:

spark.sparkContext._gateway.shutdown_callback_server()
spark.stop()

Note: use shutdown_callback_server() rather than close() — close() shuts down the entire gateway which can cause issues, while shutdown_callback_server() specifically stops the callback server that PyDeequ started.

If shutdown_callback_server() alone doesn't work for you (e.g. on AWS Glue), try:

spark.sparkContext._gateway.shutdown_callback_server()
spark.sparkContext._gateway.close()
spark.stop()

This isn't ideal — we should add a cleanup method or context manager. The root cause is in PythonCallback.__init__ which calls gateway.start_callback_server() but never provides a corresponding teardown. A proper fix would be adding a module-level or session-level cleanup utility. This remains an open enhancement.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.