upgrade snowflake driver on redash running on docker containers

[sandy124356]

someone had setup redash (v9 )on ec2 in my org long back.
When i see the docker details, i can see the drivers for snowflake were pretty old. (not even supported anymore)
Even the recent version of redash (v10) don't have latest snowflake drivers.

So I am thinking to update the snowflake driver version in requirements_ds.txt to latest version.

Since redash is running as docker container and i don't know where the docker-compose file in the ec2 machine, i am kind of struck.
Can some please help me with thoughts...

my questions, how to find where the docker-compose file is, on this ec2? (docker ps didn't gave any paths/clues)
What should i edit to make it use my defined version for snowflake driver ?
is this approach correct ? or there any other alternatives ?

[justinclift]

Ahhh. It's not always super straight forward to update the versions of dependencies, like the Snowflake driver, in Python. 😦

While the docker compose file you're after is likely under /opt/redash, when you update one thing it can have its own dependencies which in turn need updating. It's kind of a pain. 😦

Have you looked into upgrading the Redash installation as a whole to a newer version? Also, make sure to take a full backup first of the whole thing (ie complete EC2 instance, or at least the Redash database plus + env file).

Note - there have been instances where the upgrade process doesn't go as smoothly as desired the first time around, so that backup thing is really, really, really important. 😉

[sandy124356]

Thanks Justin for the info.
I appreciate that you took some time for this..

1. Have you looked into upgrading the Redash installation as a whole to a newer version? - Actually I don't see there is any latest redash version for 2024. - last version is https://github.com/getredash/redash/releases (v10.1.0 2021 Nov), but i want to use snowflake-connector-python==2.1.3 which is released in 2022. So i am skeptic even with latest upgrade of redash i wont get latest snowflake driver.

Can you please help me in this regard on how can i use latest snowflake driver ?
Actually i am not sure how to change the requirements.txt file in a existing docker image and also i am not sure if that is the only way to handle this situation.

(Yes, I now can see compose at /opt/redash)

[justinclift]

Ahhh. We have monthly snapshot images of the Redash development code which are "pretty good", but we haven't done an official release yet since the last major one back in 2022.

There's an outstanding bug that breaks embedded charts for some people that needs chasing down. When that's done then we'll be able to make a new official release.

---

That all being said, in your situation I'd probably try this:

a) Make a backup 😁
b) Follow up the "How to Upgrade" page and step through the versions one by one through to the 10.1.0 mentioned on that page.
c) Follow the upgrade process again, but this time use the preview image (generated from recent development snapshot) instead
d) Test the living heck out of the resulting Redash installation, and if something looks wrong then return to your backup and figure out a new plan? 😄

[snemmaluri-tmx]

"Test the living heck out of the resulting Redash installation, " !!! 💯 😄

So I updated my docker to 20.x from 19. as preview images are complaining while pulling.
I downloaded one of the preview tag image ( 24.08.1-dev) as there is no latest tag.
https://hub.docker.com/layers/redash/preview/24.08.1-dev/images/sha256-56a45bfefb2382aebafbd00bdbeedb7b22a1a935459b8e6980cffd9bb3a0e79f?context=explore
Finally referred it in docker-compose to see the big blow below!!

"Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application."

May be there should be an easier way to refer to latest driver of snowflake/or any while being on the old Redash version..
(Also i could not able to find the requirements_ds.txt when ssh into redash container.)

[justinclift]

Also i could not able to find the requirements_ds.txt when ssh into redash container.

Is that with one of the EC2 based Redash images?

If so, then yeah, that's expected. I don't personally use the EC2 builds, but my understanding of them is that they're a fairly standard Linux install of some sort with Redash installed under '/opt/redash` using an older version of the setup.sh script here.

That setup.sh script pretty much just installs Docker, then installs Redash using the pre-built images. It doesn't use the GitHub repository of Redash code at all.

The snowflake driver (in Python) in particular is a pain in the butt as newer versions of it have stated requirements on newer versions of Python's cryptography packages. So we have to wait for things to line up before we can upgrade it. 😬

---

With that internal server error, did you upgrade release by release (i.e. 9 to 10, 10 to preview) or did you bump up the version straight from the older release to a preview tag image?

If it was a jump from an older release directly to the preview image, then that's likely to be the problem and something to try for the next attempt. 😉

If the more proper one-version-at-a-time upgrade doesn't work, then it'll mean the internal server error problem needs investigating. It might turn out to be something simple. 😄

[sandy124356]

Hey @justinclift
So i tried this and it worked. (at least it seems to be..)
(Experimented below on latest redash/redash 10.1.0.b50633 image.)
1.After docker-compose up, I can see redash_server_1 container running.. I ssh into it, checked version of snowflake driver which is pretty old. (2.4.3).
2.took a backup of pip list.
3.upgraded the module to latest supported version (2.8.0) using pip (pip install --upgrade snowflake-connector-python==2.8.0)
4. took pip list again and compared what has changed/added
5. As expected and as you mentioned could see cryptography, filelock, typing-extensions have versions upgraded.(rest all module all same versions)
6. Updated requirements.txt inside the container with new versions of above modules.
7. updated requirements_all_ds.txt of the container with 2.8.0 version of snowflake-connector-python.
8. Created a new image out of my running conainer.
9. added a tag for it.
10. modified docker-compose.yml at /opt/redash to use my created image from step 8.
11. docker-compose up.

I can see that I am now using latest snowflake connector. Might be lucky as there are not so many modules i need to handle/upgrade.
This may not be the straight-forward way, but please let me know if you see any risks.
I will upload this image to hub, may be useful for others.
Thanks a ton for your help.

[justinclift]

Interesting approach. Not seeing anything technically wrong with that, and it sounds like it's doing what you need. 😄