Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement optional index tags in create_index and configure_index #426

Open
wants to merge 3 commits into
base: release-candidate/2025-01
Choose a base branch
from

Conversation

jhamon
Copy link
Collaborator

@jhamon jhamon commented Dec 19, 2024

Problem

Want to expose kwargs for setting and modifying index tags

Solution

The usual. The generated bits backing this feature already got created when I generated off the 2025-01 spec.

I added a small amount of logic to handle merging new and existing tags when using configure.

Usage

>>> from pinecone import Pinecone, ServerlessSpec
>>> pc = Pinecone(api_key='key')

# Create index with optional tags dictionary
>>> pc.create_index(
...   name='myindex',
...   dimension=10,
...   metric='cosine',
...   spec=ServerlessSpec(cloud='aws', region='us-west-2'),
...   vector_type='dense',
...   tags={'env': 'testing', 'author': 'jhamon'}
... )
>>> pc.describe_index(name='myindex')
{
    "name": "myindex",
    "metric": "cosine",
    "host": "myindex-dojoi3u.svc.apw5-4e34-81fa.pinecone.io",
    "spec": {
        "serverless": {
            "cloud": "aws",
            "region": "us-west-2"
        }
    },
    "status": {
        "ready": true,
        "state": "Ready"
    },
    "vector_type": "dense",
    "dimension": 10,
    "deletion_protection": "disabled",
    "tags": {
        "author": "jhamon",
        "env": "testing"
    }
}

# Update a tag value. This will be merged with existing tags.
>>> pc.configure_index(
...   name='myindex',
...   tags={'env': 'production'}
... )
>>> pc.describe_index(name='myindex')
{
    "name": "myindex",
    "metric": "cosine",
    "host": "myindex-dojoi3u.svc.apw5-4e34-81fa.pinecone.io",
    "spec": {
        "serverless": {
            "cloud": "aws",
            "region": "us-west-2"
        }
    },
    "status": {
        "ready": true,
        "state": "Ready"
    },
    "vector_type": "dense",
    "dimension": 10,
    "deletion_protection": "disabled",
    "tags": {
        "author": "jhamon",
        "env": "production"
    }
}

# Remove a tag by sending empty string value
>>> pc.configure_index(
...   name='myindex',
...   tags={'author': ''}
... )
>>> pc.describe_index(name='myindex')
{
    "name": "myindex",
    "metric": "cosine",
    "host": "myindex-dojoi3u.svc.apw5-4e34-81fa.pinecone.io",
    "spec": {
        "serverless": {
            "cloud": "aws",
            "region": "us-west-2"
        }
    },
    "status": {
        "ready": true,
        "state": "Ready"
    },
    "vector_type": "dense",
    "dimension": 10,
    "deletion_protection": "disabled",
    "tags": {
        "env": "production"
    }
}

Type of Change

  • New feature (non-breaking change which adds functionality)

@jhamon jhamon marked this pull request as ready for review December 19, 2024 15:51
@jhamon jhamon changed the title Implement index tags in create and configure index Implement optional index tags in create_index and configure_index Dec 19, 2024
Base automatically changed from jhamon/sparse-indexes to release-candidate/2025-01 December 19, 2024 16:27
Copy link
Contributor

@austin-denoble austin-denoble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

only nit is we may want to include a minimal amount of documentation about needing to use "" to delete / clear a tag

@@ -280,17 +288,31 @@ def configure_index(
replicas: Optional[int] = None,
pod_type: Optional[str] = None,
deletion_protection: Optional[Literal["enabled", "disabled"]] = None,
tags: Optional[Dict[str, str]] = None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Should the docstring in the interface be updated?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, thanks.

@aulorbe
Copy link

aulorbe commented Dec 28, 2024

This might be an unnecessary question, but why do you have to merge existing tags? At least when I hit this endpoint in Node, existing tags are already merged (i.e. a configure req (containing a new tag) that is sent to an index w/existing tags already merges the new and existing tags).

Previous to this PR, were tags being overwritten when updated when you hit this endpoint in Python?

):
"""This method is used to scale configuration fields for your pod-based Pinecone index.

:param: name: the name of the Index
:param: replicas: the desired number of replicas, lowest value is 0.
:param: pod_type: the new pod_type for the index. To learn more about the
available pod types, please see [Understanding Indexes](https://docs.pinecone.io/docs/indexes)

:param: deletion_protection: If set to 'enabled', the index cannot be deleted. If 'disabled', the index can be deleted.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I'd add that enabled is the default here.

@jhamon
Copy link
Collaborator Author

jhamon commented Jan 7, 2025

This might be an unnecessary question, but why do you have to merge existing tags? At least when I hit this endpoint in Node, existing tags are already merged (i.e. a configure req (containing a new tag) that is sent to an index w/existing tags already merges the new and existing tags).

@aulorbe We're doing this to workaround some odd behavior in the generated python code. For example, when we first implemented deletion_protection in this configure method we saw that deletion protection was getting switched off (to the default value of disabled) if the user wanted to make some other unrelated change, like to scale the number of replicas, without specifying a change for the deletion_protection field. This was happening because the generated request object uses the default value (disabled) when none is specified. In order to avoid overwriting properties with default values on updates, we fetch and merge current values with the new values provided by the user.

Tags are a similar story. I was told the way people are supposed to delete things is by passing {"key": ""} to delete a key from tags. I wanted to test that and used merge to make explicit what is being sent in my tests rather than rely on any sort of implicit behavior around None or {}. Perhaps it's not strictly necessary to get a correct result in this case, but it doesn't add any performance cost to do it this way since I already have to fetch the current state to get the deletion_protection value.

As for whether this is necessary or not in other SDKs, I think it really depends on how those language generators translate the spec into code. You'll have to do your own testing to make sure empty/default values are not doing unintended things when a user is only trying to modify other fields.

Previous to this PR, were tags being overwritten when updated when you hit this endpoint in Python?

This is the first time we've had tags in the python SDK. So there's no precedent of doing it any other way.

@aulorbe
Copy link

aulorbe commented Jan 7, 2025

I see, thanks @jhamon !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants