Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added worker for Snowflake's Snowpark Container Services #16393

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

bjorhn
Copy link
Contributor

@bjorhn bjorhn commented Dec 15, 2024

This PR adds a new worker to the prefect-snowflake integration. The worker can run service jobs in Snowflake's Snowpark Container Services. Closes #15674

A few things worth mentioning:

  • Snowflake's Python API currently doesn't support running async service jobs, so I'm temporarily running jobs using SQL commands instead. I will submit a PR to resolve this as soon as Snowflake adds the required features to their Python API.
  • Most of the code was hacked together based on the official "custom worker" tutorial and copying things over from the Azure and Kubernetes workers. The same goes for the unit tests.
  • Since a lot of the structure was copied over, let me know if I misunderstood how something should be done. For example, am I using run_sync_in_worker_thread correctly?
  • I don't have a lot of experience creating unit tests in Python, so I've done the bare minimum with regards to unit testing.

I don't know the best way to test the worker after moving the modules over from our company repo to the Prefect repo. Perhaps someone could test it for me, or leave instructions in a comment. The code should be the same as it were before, it's just some import paths that have changed.

There are two bugs I've bumped into along the way which I don't think are caused by my code. I don't know if these issues only affect custom workers, but I think so:

@github-actions github-actions bot added the enhancement An improvement of an existing feature label Dec 15, 2024
Copy link
Contributor

This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment.

@bjorhn
Copy link
Contributor Author

bjorhn commented Jan 11, 2025

Is anything else required for this to be merged?

@desertaxle
Copy link
Member

Is anything else required for this to be merged?

Hey @bjorhn! Apologies for the delay in review. Could you share some instructions for how to test out this worker? I haven't used Snowpark Container Service before, so some guides or documentation would be very helpful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New worker for running job services in Snowpark Container Services
2 participants