Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improvement: Prevent ZedToken's from being used cross-datastore #1541

Open
josephschorr opened this issue Sep 19, 2023 · 2 comments · May be fixed by #1723
Open

Improvement: Prevent ZedToken's from being used cross-datastore #1541

josephschorr opened this issue Sep 19, 2023 · 2 comments · May be fixed by #1723
Assignees
@josephschorr
Labels
area/api v1 Affects the v1 API area/datastore Affects the storage system kind/tech debt Addresses legacy code/decisions priority/3 low This would be nice to have

Comments

@josephschorr
Copy link
Member

josephschorr commented Sep 19, 2023 •
edited
Loading

Right now, ZedTokens simply encode the datastore-provided revision information, with no additional information as to the kind of datastore used. This can result in weird behavior if a ZedToken from one SpiceDB cluster is used with a completely different cluster, or the datastore has been swapped out.

We should instead encode the datastore kind (and perhaps a unique ID) into the ZedToken, and either reject all such tokens, or automatically switch to full consistency (likely the former)
@josephschorr josephschorr added priority/3 low This would be nice to have area/api v1 Affects the v1 API area/datastore Affects the storage system kind/tech debt Addresses legacy code/decisions labels Sep 19, 2023
@MichaelCordingley
Copy link

Please consider that stored tokens can be scattered across multiple services managed by multiple teams. Somehow invalidating all of those tokens and keeping new ones from saving during a datastore change can be a herculean effort. I'd like to suggest that SpiceDB's response to tokens that don't match its current datastore be configurable.

To wit. If I were to have this kind of control and have to migrate data stores, I'd perform the following steps:

  1. Configure SpiceDB to switch to full consistency when it sees a legacy token.
  2. Cut over data stores.
  3. Wait thirty seconds for the SpiceDB caches to become fresher than extant pre-migration tokens.
  4. Reconfigure SpiceDB to instead consider all legacy tokens to be far enough in the past that the current cache should be used instead.

I may opt for intentional downtime and instead do:

  1. Start my maintenance window and shut off access.
  2. Configure SpiceDB to consider all legacy tokens to be far enough in the past that the current cache should be used instead.
  3. Cut over data stores.
  4. End my maintenance window.

@josephschorr
Copy link
Member Author

I think making it configurable is reasonable

josephschorr added a commit to josephschorr/spicedb that referenced this issue Jan 29, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
ae06b46 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
@josephschorr josephschorr self-assigned this Jan 29, 2024
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jan 29, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
58d56ef 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jan 29, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
d3b259f 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jan 29, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
64dda15 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jan 30, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
8f8f83c 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Mar 13, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
8c87502 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Mar 13, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
52895fe 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Mar 13, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
9bb170d 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Apr 1, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
d15a06b 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Apr 16, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
ba65181 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Apr 22, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
160e514 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jul 1, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
c176cb2 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jul 1, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
da44237 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
@josephschorr josephschorr linked a pull request Jul 9, 2024 that will close this issue
Open
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jul 11, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
33c21c5 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jul 12, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
f120b8d 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jul 30, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
dfb450a 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Aug 6, 2024
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
69477e0 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
josephschorr added a commit to josephschorr/spicedb that referenced this issue Jan 6, 2025
@josephschorr
Setup defined (and configurable) behavior if a ZedToken from
9b3ffc0 
an older datastore is used

All ZedTokens are now minted with the datastore's unique ID included
in the ZedToken and that ID is checked when the ZedToken is decoded.

In scenarios where the datastore ID does not match, either an error is
raised (watch, at_exact_snapshot) or configurable behavior is used
(at_least_as_fresh)

Fixes authzed#1541
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@josephschorr josephschorr

Labels
area/api v1 Affects the v1 API area/datastore Affects the storage system kind/tech debt Addresses legacy code/decisions priority/3 low This would be nice to have
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Proper handling of incompatible zedtokens josephschorr/spicedb
2 participants
@josephschorr @MichaelCordingley