feat(context-agent): move Deep Cody out of model dropdown (#6513)

Adds on #6513 CLOSE https://linear.app/sourcegraph/issue/CODY-4535 https://linear.app/sourcegraph/issue/CODY-4563 https://linear.app/sourcegraph/issue/CODY-4539 Building blocks of https://www.loom.com/share/16b55f92368844b4b766083ed4d30091 This PR introduces a new UI-based approach for managing Cody's agent capabilities, replacing the previous VS Code settings configuration. It also moves Deep Cody out of the model dropdown into its own Agent Settings component ## Key Changes 1. New Agent UI Controls: - Added `ToolboxButton` component in chat input area with: - Bot icon indicating agent status (active/inactive) - Green pulse animation for active state - Popover menu for settings - Dedicated switches for agent and terminal features 2. Deep Cody Architecture Improvements: - Simplified agent ID for Deep Cody to 'deep-cody' from previous ModelRef format - Added support for model selection between default chat model and Claude 3.5 Haiku - Moved shell context permission to UI toggle with warning message - Enhanced context handling with deduplication for repeated queries 3. API & Type Updates: - Added `toolboxSettings` and `updateToolboxSettings` to WebView API - Updated `AgentToolboxSettings` type to use string-based agent ID - Enhanced message types to support agent attribution - Added feature flag for chat model selection 4. Code Organization: - Refactored `CodyToolProvider` into namespace pattern - Improved tool factory initialization and management - Enhanced test coverage for new UI components - Removed deprecated VS Code settings 5. Default Model - Defaulted to use 3.5 Haiku as the reflection model - When the `ContextAgentDefaultChatModel` feature flag is enabled on the instance, reflection model will be set to Sonnet instead 6. Merge CodyChatAgent with DeepCodyAgent. 7. Introduce ToolboxManager and AgentToolboxSettings - Added a `ToolboxManager` class to centralize the management of agent toolbox settings, including the terminal/shell context. - Introduced the `AgentToolboxSettings` interface to represent the user's preferences for the agent and terminal context. - Integrated the `ToolboxManager` into the `ChatController` to update the user's terminal context setting based on the configuration. - Persisted the agent and terminal context settings in the local storage for user-specific preferences. - ToolboxManager settings for terminal context gets updated on user config changes 8. Improve prompt and response format for context review - Enhance the review prompt to provide clearer instructions and the expected response format. - Add new tags for context and answers to standardize the response structure. - Provide examples of valid and invalid responses to guide the user. - Update the prompt to emphasize the importance of only including the expected tags in the response. - Integrate the ToolboxManager into the ChatController to update the user's terminal context setting based on the configuration. - Persist the agent and terminal context settings in the local storage for user-specific preferences. 9. Improve context retrieval for deep-cody / context agent - send context to webview for display before sending the request - update context retrieval to only include validated context items that match the end of the path - post empty message in progress when processing steps are updated ## TO-DO Will be continued in follow-ups: - [ ] Update Deep Cody Notice / upsell with the new changes - [ ] Remove Deep Cody rate limit - [ ] Update Agentic context UI - [ ] Update New toolbox setting UI ## Test plan  Updated all existing unit tests and added new unit tests to cover all the Deep Cody components. Deep Cody should work as it does currently: Example Question: `how many files are there in the root of my codebase? Find where PromptSelectField is defined in the codebase` ![image](https://github.com/user-attachments/assets/3cbfa27a-b66c-453a-ad59-db8abffc4005) In UI, enabling the Terminal Context Agent would enable terminal context ![image](https://github.com/user-attachments/assets/72545f53-0aac-48cd-b585-6a066f6bac0a) In UI, disabling the Terminal Context Agent would disable terminal context ![image](https://github.com/user-attachments/assets/5b5d27ec-e5a5-49b5-a776-e4332d8d8709) 1. Agent Controls: - [x] Verify bot icon appears in chat input - [x] Check icon states (active/inactive) - [x] Confirm green pulse animation when active - [x] Test settings popover interaction 2. Settings Management: - [x] Enable/disable agent via toggle - [x] Toggle terminal access (when available) - [x] Verify warning message for terminal access - [x] Confirm settings persist across sessions 3. Chat Experience: - [x] Test chat with agent enabled/disabled - [x] Verify model selection works correctly - [x] Check context gathering improvements - [x] Confirm terminal integration (when enabled) 4. Settings Transition: - [x] Verify existing setting is not respected and is marked as deprecated - [x] The Terminal context is disabled by default in the new UI component - [x] Test default states as PLG and Enterprise users - [x] Check settings sync behavior ## Changelog  --------- Co-authored-by: Valery Bugakov <[email protected]>
sourcegraph · Jan 7, 2025 · fabed61 · fabed61
1 parent 77df8d2
commit fabed61
Show file tree

Hide file tree

Showing 42 changed files with 1,441 additions and 682 deletions.
diff --git a/...ib/src/main/kotlin/com/sourcegraph/cody/agent/protocol_generated/SerializedChatMessage.kt b/...ib/src/main/kotlin/com/sourcegraph/cody/agent/protocol_generated/SerializedChatMessage.kt
@@ -13,6 +13,7 @@ data class SerializedChatMessage(
   val intent: IntentEnum? = null, // Oneof: search, chat, edit, insert
   val manuallySelectedIntent: ManuallySelectedIntentEnum? = null, // Oneof: search, chat, edit, insert
   val search: Any? = null,
+  val agent: String? = null,
   val processes: List<ProcessingStep>? = null,
   val subMessages: List<SubMessage>? = null,
 ) {

diff --git a/lib/shared/src/chat/transcript/index.ts b/lib/shared/src/chat/transcript/index.ts
@@ -35,6 +35,7 @@ export function serializeChatMessage(chatMessage: ChatMessage): SerializedChatMe
         manuallySelectedIntent: chatMessage.manuallySelectedIntent,
         search: chatMessage.search,
         processes: chatMessage.processes,
+        agent: chatMessage.agent,
         subMessages: chatMessage.subMessages,
     }
 }
diff --git a/lib/shared/src/chat/transcript/messages.ts b/lib/shared/src/chat/transcript/messages.ts
@@ -45,6 +45,7 @@ export interface ChatMessage extends Message {
     intent?: 'search' | 'chat' | 'edit' | 'insert' | undefined | null
     manuallySelectedIntent?: 'search' | 'chat' | 'edit' | 'insert' | undefined | null
     search?: ChatMessageSearch | undefined | null
+    agent?: string
     processes?: ProcessingStep[] | undefined | null
 
     /**
@@ -115,6 +116,7 @@ export interface SerializedChatMessage {
     intent?: ChatMessage['intent']
     manuallySelectedIntent?: ChatMessage['manuallySelectedIntent']
     search?: ChatMessage['search']
+    agent?: string
     processes?: ProcessingStep[] | undefined | null
     subMessages?: SubMessage[]
 }

diff --git a/lib/shared/src/configuration.ts b/lib/shared/src/configuration.ts
@@ -87,7 +87,6 @@ interface RawClientConfiguration {
     commandCodeLenses: boolean
 
     // Deep Cody
-    agenticContextExperimentalShell?: boolean
     agenticContextExperimentalOptions?: AgenticContextConfiguration
 
     //#region Autocomplete
@@ -481,3 +480,23 @@ export interface FireworksCodeCompletionParams {
     languageId: string
     user: string | null
 }
+
+export interface AgentToolboxSettings {
+    /**
+     * The agent that user has currently enabled.
+     */
+    agent?: {
+        /**
+         * The name of the agent that user has currently enabled. E.g. "deep-cody"
+         */
+        name?: string
+    }
+    /**
+     * Whether the user has enabled terminal context.
+     * Defaulted to undefined if shell context is not enabled by site admin via feature flag.
+     */
+    shell?: {
+        enabled: boolean
+        error?: string
+    }
+}
diff --git a/lib/shared/src/experimentation/FeatureFlagProvider.ts b/lib/shared/src/experimentation/FeatureFlagProvider.ts
@@ -107,6 +107,9 @@ export enum FeatureFlag {
     /** Enable Shell Context for Deep Cody */
     DeepCodyShellContext = 'deep-cody-shell-context',
 
+    /** Whether Context Agent (Deep Cody) should use the default chat model or 3.5 Haiku */
+    ContextAgentDefaultChatModel = 'context-agent-use-default-chat-model',
+
     /** Enable Rate Limit for Deep Cody */
     DeepCodyRateLimitBase = 'deep-cody-experimental-rate-limit',
     DeepCodyRateLimitMultiplier = 'deep-cody-experimental-rate-limit-multiplier',

diff --git a/lib/shared/src/misc/rpc/webviewAPI.ts b/lib/shared/src/misc/rpc/webviewAPI.ts
@@ -1,5 +1,11 @@
 import { Observable } from 'observable-fns'
-import type { AuthStatus, ModelsData, ResolvedConfiguration, UserProductSubscription } from '../..'
+import type {
+    AgentToolboxSettings,
+    AuthStatus,
+    ModelsData,
+    ResolvedConfiguration,
+    UserProductSubscription,
+} from '../..'
 import type { SerializedPromptEditorState } from '../..'
 import type { ChatMessage, UserLocalHistory } from '../../chat/transcript/messages'
 import type { ContextItem, DefaultContext } from '../../codebase-context/messages'
@@ -104,6 +110,15 @@ export interface WebviewToExtensionAPI {
      * The current user's product subscription information (Cody Free/Pro).
      */
     userProductSubscription(): Observable<UserProductSubscription | null>
+
+    /**
+     * The current user's toolbox settings.
+     */
+    toolboxSettings(): Observable<AgentToolboxSettings | null>
+    /**
+     *  Update the current user's toolbox settings.
+     */
+    updateToolboxSettings(settings: AgentToolboxSettings): Observable<void>
 }
 
 export function createExtensionAPI(
@@ -138,6 +153,8 @@ export function createExtensionAPI(
         transcript: proxyExtensionAPI(messageAPI, 'transcript'),
         userHistory: proxyExtensionAPI(messageAPI, 'userHistory'),
         userProductSubscription: proxyExtensionAPI(messageAPI, 'userProductSubscription'),
+        toolboxSettings: proxyExtensionAPI(messageAPI, 'toolboxSettings'),
+        updateToolboxSettings: proxyExtensionAPI(messageAPI, 'updateToolboxSettings'),
     }
 }
 

diff --git a/lib/shared/src/models/sync.test.ts b/lib/shared/src/models/sync.test.ts
@@ -462,70 +462,6 @@ describe('syncModels', () => {
         }
     )
 
-    it('sets DeepCody as default chat model when feature flag is enabled', async () => {
-        const serverSonnet: ServerModel = {
-            modelRef: 'anthropic::unknown::sonnet',
-            displayName: 'Sonnet',
-            modelName: 'anthropic.claude-3-5-sonnet',
-            capabilities: ['chat'],
-            category: 'balanced' as ModelCategory,
-            status: 'stable',
-            tier: 'enterprise' as ModelTier,
-            contextWindow: {
-                maxInputTokens: 9000,
-                maxOutputTokens: 4000,
-            },
-        }
-
-        const SERVER_MODELS: ServerModelConfiguration = {
-            schemaVersion: '0.0',
-            revision: '-',
-            providers: [],
-            models: [serverSonnet],
-            defaultModels: {
-                chat: serverSonnet.modelRef,
-                fastChat: serverSonnet.modelRef,
-                codeCompletion: serverSonnet.modelRef,
-            },
-        }
-
-        const mockFetchServerSideModels = vi.fn(() => Promise.resolve(SERVER_MODELS))
-        vi.mocked(featureFlagProvider).evaluatedFeatureFlag.mockReturnValue(Observable.of(true))
-
-        const result = await firstValueFrom(
-            syncModels({
-                resolvedConfig: Observable.of({
-                    configuration: {},
-                    clientState: { modelPreferences: {} },
-                } satisfies PartialDeep<ResolvedConfiguration> as ResolvedConfiguration),
-                authStatus: Observable.of(AUTH_STATUS_FIXTURE_AUTHED),
-                configOverwrites: Observable.of(null),
-                clientConfig: Observable.of({
-                    modelsAPIEnabled: true,
-                } satisfies Partial<CodyClientConfig> as CodyClientConfig),
-                fetchServerSideModels_: mockFetchServerSideModels,
-                userProductSubscription: Observable.of({ userCanUpgrade: true }),
-            }).pipe(skipPendingOperation())
-        )
-
-        const storage = new TestLocalStorageForModelPreferences()
-        modelsService.setStorage(storage)
-        mockAuthStatus(AUTH_STATUS_FIXTURE_AUTHED)
-        expect(storage.data?.[AUTH_STATUS_FIXTURE_AUTHED.endpoint]!.selected.chat).toBe(undefined)
-        vi.spyOn(modelsService, 'modelsChanges', 'get').mockReturnValue(Observable.of(result))
-
-        // Check if Deep Cody model is in the primary models list.
-        expect(result.primaryModels.some(model => model.id.includes('deep-cody'))).toBe(true)
-
-        // Deep Cody should not replace the default chat / edit model.
-        expect(result.preferences.defaults.chat?.includes('deep-cody')).toBe(false)
-        expect(result.preferences.defaults.edit?.includes('deep-cody')).toBe(false)
-
-        // preference should not be affected and remains unchanged as this is handled in a later step.
-        expect(result.preferences.selected.chat).toBe(undefined)
-        expect(storage.data?.[AUTH_STATUS_FIXTURE_AUTHED.endpoint]!.selected.chat).toBe(undefined)
-    })
-
     describe('model selection based on user tier and feature flags', () => {
         const serverHaiku: ServerModel = {
             modelRef: 'anthropic::unknown::claude-3-5-haiku',

diff --git a/lib/shared/src/models/sync.ts b/lib/shared/src/models/sync.ts
@@ -27,7 +27,7 @@ import { RestClient } from '../sourcegraph-api/rest/client'
 import type { UserProductSubscription } from '../sourcegraph-api/userProductSubscription'
 import { CHAT_INPUT_TOKEN_BUDGET } from '../token/constants'
 import { isError } from '../utils'
-import { TOOL_CODY_MODEL, getExperimentalClientModelByFeatureFlag } from './client'
+import { TOOL_CODY_MODEL } from './client'
 import { type Model, type ServerModel, createModel, createModelFromServerModel } from './model'
 import type {
     DefaultsAndUserPreferencesForEndpoint,
@@ -212,12 +212,7 @@ export function syncModels({
                                         enableToolCody
                                     ).pipe(
                                         switchMap(
-                                            ([
-                                                hasEarlyAccess,
-                                                hasDeepCodyFlag,
-                                                defaultToHaiku,
-                                                enableToolCody,
-                                            ]) => {
+                                            ([hasEarlyAccess, isDeepCodyEnabled, defaultToHaiku]) => {
                                                 // TODO(sqs): remove waitlist from localStorage when user has access
                                                 const isOnWaitlist = config.clientState.waitlist_o1
                                                 if (isDotComUser && (hasEarlyAccess || isOnWaitlist)) {
@@ -238,40 +233,11 @@ export function syncModels({
                                                         }
                                                     )
                                                 }
-
-                                                // Replace user's current sonnet model with deep-cody model.
-                                                const sonnetModel = data.primaryModels.find(m =>
-                                                    m.id.includes('sonnet')
-                                                )
-                                                // DEEP CODY is enabled for all PLG users.
-                                                // Enterprise users need to have the feature flag enabled.
-                                                const isDeepCodyEnabled =
-                                                    (isDotComUser && !isCodyFreeUser) || hasDeepCodyFlag
-                                                if (
-                                                    isDeepCodyEnabled &&
-                                                    sonnetModel &&
-                                                    // Ensure the deep-cody model is only added once.
-                                                    !data.primaryModels.some(m =>
-                                                        m.id.includes('deep-cody')
-                                                    )
-                                                ) {
-                                                    const DEEPCODY_MODEL =
-                                                        getExperimentalClientModelByFeatureFlag(
-                                                            FeatureFlag.DeepCody
-                                                        )!
+                                                if (isDeepCodyEnabled && enableToolCody) {
                                                     data.primaryModels.push(
-                                                        ...maybeAdjustContextWindows([
-                                                            DEEPCODY_MODEL,
-                                                        ]).map(createModelFromServerModel)
+                                                        createModelFromServerModel(TOOL_CODY_MODEL)
                                                     )
-
-                                                    if (enableToolCody) {
-                                                        data.primaryModels.push(
-                                                            createModelFromServerModel(TOOL_CODY_MODEL)
-                                                        )
-                                                    }
                                                 }
-
                                                 // set the default model to Haiku for free users
                                                 if (isDotComUser && isCodyFreeUser && defaultToHaiku) {
                                                     const haikuModel = data.primaryModels.find(m =>

diff --git a/lib/shared/src/prompt/prompt-mixin.ts b/lib/shared/src/prompt/prompt-mixin.ts
@@ -43,17 +43,16 @@ export class PromptMixin {
             mixins.push(PromptMixin.hedging)
         }
 
-        // Handle Deep Cody specific prompts
-        const isDeepCodyEnabled = modelID?.includes('deep-cody')
-        if (isDeepCodyEnabled && !newMixins.length) {
+        // Handle Agent specific prompts
+        if (humanMessage.agent === 'deep-cody' && !newMixins.length) {
             mixins.push(new PromptMixin(HEDGES_PREVENTION.concat(DEEP_CODY)))
         }
 
         // Add new mixins to the list of mixins to be prepended to the next human message.
         mixins.push(...newMixins)
 
         const prompt = PromptMixin.buildPrompt(mixins)
-        return PromptMixin.mixedMessage(humanMessage, prompt, mixins, isDeepCodyEnabled)
+        return PromptMixin.mixedMessage(humanMessage, prompt, mixins)
     }
 
     private static buildPrompt(mixins: PromptMixin[]): PromptString {
@@ -67,17 +66,16 @@ export class PromptMixin {
     private static mixedMessage(
         humanMessage: ChatMessage,
         prompt: PromptString,
-        mixins: PromptMixin[],
-        isDeepCodyEnabled = false
+        mixins: PromptMixin[]
     ): ChatMessage {
         if (!mixins.length || !humanMessage.text) {
             return humanMessage
         }
 
-        if (isDeepCodyEnabled) {
+        if (humanMessage.agent === 'deep-cody' && prompt.includes('{{USER_INPUT_TEXT}}')) {
             return {
                 ...humanMessage,
-                text: ps`${prompt}\n\n[QUESTION]\n`.concat(humanMessage.text),
+                text: prompt.replace('{{USER_INPUT_TEXT}}', humanMessage.text),
             }
         }
 

diff --git a/vscode/package.json b/vscode/package.json
@@ -1270,8 +1270,9 @@
         },
         "cody.agentic.context.experimentalShell": {
           "type": "boolean",
-          "markdownDescription": "Enable Agents like Deep Cody to autonomously execute shell commands in your environment for context. Enable with caution as mistakes are possible.",
-          "default": false
+          "value": false,
+          "markdownDeprecationMessage": "**Deprecated** Enable Agents like Deep Cody to autonomously execute shell commands in your environment for context. Enable with caution as mistakes are possible.",
+          "deprecationMessage": "Deprecated. Now configurable via UI."
         },
         "cody.agentic.context.experimentalOptions": {
           "type": "object",