Migrate to the task processing API

[julien-nc]

Translation and SpeechToText backend APIs are deprecated. Those features are now included in the task processing API (since 30).

The old APIs will stay a few more NC major version. The old SpeechToText API can now use the new providers (for TaskProcessing) so there is no rush to migrate.

The providers for the Translate API and the translation providers for the TaskProcessing API can be installed side by side so there is no rush to migrate there either.

Translation

You can use the assistant's UI to run translation tasks in the UI. If the assistant app is enabled, the OCA.Assistant.openAssistantForm function should be available.

if (OCA.Assistant.openAssistantForm) {
	OCA.Assistant.openAssistantForm({
		appId: 'spreed',
		customId: 'message translation',
		taskType: 'core:text2text:translate',
		inputs: {
			input: 'the content of the message',
		},
		closeOnResult: false,
	}).then(task => {
		if (task.status === 'STATUS_SUCCESSFUL') {
			console.debug('assistant result task output', task.output.output)
		} else {
			console.debug('assistant result task', task)
		}
	})
}

The promise will resolve if the task succeeds, fails or is scheduled for later by the user. The promise result is the task object.
The closeOnResult parameter of OCA.Assistant.openAssistantForm decides if the assistant is closed when the task succeeds of fails. It can be false to stay close to the current behaviour of the translate modal in Talk. The user sees the result in the assistant and there is a "copy" button. The user can then close the assistant modal.

SpeechToText

Transcription can be done with the core:audio2text task type of the taskProcessing API. More details on how to run such task in the backend in the Transcribe section of nextcloud/assistant#114

[nickvergessen]

@julien-nc we have a bit of a problem here.

Translating chat messages

We need OCS APIs as our mobile and desktop clients are calling it, and they should "respond" with it and not be delegated to a background job (No one will wait 5 minutes on the translation of a chat message).

Transcription of call recordings

Can be done in a background job, should be fine (we do that now as well as far as I know)

[nickvergessen]

Also the API endpoints in https://github.com/nextcloud/server/blob/bc5c0262af3cd375620d6534353a3842149ad6ab/core/Controller/TranslationApiController.php are not marked as @deprecated

[julien-nc]

No one will wait 5 minutes on the translation of a chat message

If the provider is an exApp, it will process tasks ASAP, no delay. If the provider is a Php app and occ background-job:worker "OC\TaskProcessing\SynchronousBackgroundJob" is running, no delay either.

Once the task is scheduled, the clients can poll it with ocs/v2.php/taskprocessing/task/TASK_ID. That's what the assistant does in the frontend. No more blocking request as it could be too long and be killed but also it blocks a Php runner while waiting.

[nickvergessen]

So instead of getting a string returned the clients shall DOS the server.
The feature still breaks for existing clients.

[julien-nc]

We can also keep the providers for the old APIs in integration_openai and the features in Talk are not broken.

[nickvergessen]

• 🟢 translate https://apps.nextcloud.com/apps/translate
• 🔴 integration_openai https://apps.nextcloud.com/apps/integration_openai

I will check with Andy next week what to do.

[julien-nc]

Two things should make it more convenient:

• The TextProcessing and SpeechToText APIs are now forward compatible with providers. New TaskProcessing providers can be used by the TextProcessing API (for FreePromptTaskType, HeadlineTaskType, SummaryTaskType and TopicsTaskType because they have exact matches in the new API) and the SpeechToText API. This means you will benefit from new providers while using the old APIs.
• The TaskProcessing manager now has a runTask method to run a task synchronously. This should make the migration easier.

All this is in stable30 already.

The providers for the Translate API and the TaskProcessing API are implemented in different apps so you can keep using the Translate API as long as you want.

[nickvergessen]

• Call recording transcriptions:
  Was done when introducing the summary: feat(AI-call-summary): Automatically summarize call transcript #13807
• Chat message translations:
  Kind of problematic, as it needs adjustments in the clients as well

Translations

• Add new capability if task processing for translation is there feat(translations): Expose task-processing translation options #14068
• Fetch GET /ocs/v2.php/taskprocessing/tasktypes (?) read the input shapes things
• Create From and To language sets from the offered ENUMs
• Use task processing API directly to generate the task and await the translation https://docs.nextcloud.com/server/latest/developer_manual/client_apis/OCS/ocs-taskprocessing-api.html

(Sample with Groq as intergration_openai App):

curl -k 'https://nextcloud31.local/ocs/v2.php/taskprocessing/tasktypes?format=json' -H 'OCS-APIRequest: true' | jq '.ocs.data.types["core:text2text:translate"]'

{
  "name": "Translate",
  "description": "Translate text from one language to another",
  "optionalInputShape": [
    {
      "name": "Maximum output words",
      "description": "The maximum number of words/tokens that can be generated in the completion.",
      "type": "Number"
    },
    {
      "name": "Model",
      "description": "The model used to generate the completion",
      "type": "Enum"
    }
  ],
  "inputShapeEnumValues": [
    [
      {
        "name": "Detect language",
        "value": "detect_language"
      },
      {
        "name": "English (US)",
        "value": "en"
      },
      {
        "name": "Español (España)",
        "value": "es"
      },
      {
        "name": "Français",
        "value": "fr"
      },
      {
        "name": "Deutsch (Persönlich: Du)",
        "value": "de"
      },
      {
        "name": "Deutsch (Förmlich: Sie)",
        "value": "de_DE"
      },
      {
        "name": "日本語 (Japanese)",
        "value": "ja"
      },
      {
        "name": "اللغة العربية",
        "value": "ar"
      },
      {
        "name": "Русский",
        "value": "ru"
      },
      {
        "name": "Nederlands",
        "value": "nl"
      },
      {
        "name": "Italiano",
        "value": "it"
      },
      {
        "name": "Português Brasileiro",
        "value": "pt_BR"
      },
      {
        "name": "Português",
        "value": "pt_PT"
      },
      {
        "name": "Dansk",
        "value": "da"
      },
      {
        "name": "Svenska",
        "value": "sv"
      },
      {
        "name": "Türkçe",
        "value": "tr"
      },
      {
        "name": "简体中文",
        "value": "zh_CN"
      },
      {
        "name": "한국어",
        "value": "ko"
      },
      {
        "name": "Asturianu",
        "value": "ast"
      },
      {
        "name": "Bahasa Indonesia",
        "value": "id"
      },
      {
        "name": "Brezhoneg",
        "value": "br"
      },
      {
        "name": "Català",
        "value": "ca"
      },
      {
        "name": "Eesti",
        "value": "et_EE"
      },
      {
        "name": "English (British English)",
        "value": "en_GB"
      },
      {
        "name": "Español (Argentina)",
        "value": "es_AR"
      },
      {
        "name": "Español (Ecuador)",
        "value": "es_EC"
      },
      {
        "name": "Español (México)",
        "value": "es_MX"
      },
      {
        "name": "Esperanto",
        "value": "eo"
      },
      {
        "name": "Euskara",
        "value": "eu"
      },
      {
        "name": "Galego",
        "value": "gl"
      },
      {
        "name": "Hrvatski",
        "value": "hr"
      },
      {
        "name": "Latviešu",
        "value": "lv"
      },
      {
        "name": "Lietuvių",
        "value": "lt_LT"
      },
      {
        "name": "Magyar",
        "value": "hu"
      },
      {
        "name": "Norsk bokmål",
        "value": "nb"
      },
      {
        "name": "Occitan",
        "value": "oc"
      },
      {
        "name": "Polski",
        "value": "pl"
      },
      {
        "name": "Română",
        "value": "ro"
      },
      {
        "name": "Slovenčina",
        "value": "sk"
      },
      {
        "name": "Slovenščina",
        "value": "sl"
      },
      {
        "name": "Tiếng Việt",
        "value": "vi"
      },
      {
        "name": "sardu",
        "value": "sc"
      },
      {
        "name": "suomi",
        "value": "fi"
      },
      {
        "name": "Íslenska",
        "value": "is"
      },
      {
        "name": "Čeština",
        "value": "cs"
      },
      {
        "name": "Ελληνικά",
        "value": "el"
      },
      {
        "name": "Български",
        "value": "bg"
      },
      {
        "name": "Македонски",
        "value": "mk"
      },
      {
        "name": "Српски",
        "value": "sr"
      },
      {
        "name": "Українська",
        "value": "uk"
      },
      {
        "name": "עברית",
        "value": "he"
      },
      {
        "name": "ئۇيغۇرچە",
        "value": "ug"
      },
      {
        "name": "فارسى",
        "value": "fa"
      },
      {
        "name": "ไทย",
        "value": "th"
      },
      {
        "name": "ຂີ້ຕົວະ",
        "value": "lo"
      },
      {
        "name": "ქართული ენა",
        "value": "ka"
      },
      {
        "name": "正體中文（臺灣）",
        "value": "zh_TW"
      },
      {
        "name": "正體中文（香港）",
        "value": "zh_HK"
      },
      {
        "name": "ga",
        "value": "ga"
      }
    ],
    [
      {
        "name": "English (US)",
        "value": "en"
      },
      {
        "name": "Español (España)",
        "value": "es"
      },
      {
        "name": "Français",
        "value": "fr"
      },
      {
        "name": "Deutsch (Persönlich: Du)",
        "value": "de"
      },
      {
        "name": "Deutsch (Förmlich: Sie)",
        "value": "de_DE"
      },
      {
        "name": "日本語 (Japanese)",
        "value": "ja"
      },
      {
        "name": "اللغة العربية",
        "value": "ar"
      },
      {
        "name": "Русский",
        "value": "ru"
      },
      {
        "name": "Nederlands",
        "value": "nl"
      },
      {
        "name": "Italiano",
        "value": "it"
      },
      {
        "name": "Português Brasileiro",
        "value": "pt_BR"
      },
      {
        "name": "Português",
        "value": "pt_PT"
      },
      {
        "name": "Dansk",
        "value": "da"
      },
      {
        "name": "Svenska",
        "value": "sv"
      },
      {
        "name": "Türkçe",
        "value": "tr"
      },
      {
        "name": "简体中文",
        "value": "zh_CN"
      },
      {
        "name": "한국어",
        "value": "ko"
      },
      {
        "name": "Asturianu",
        "value": "ast"
      },
      {
        "name": "Bahasa Indonesia",
        "value": "id"
      },
      {
        "name": "Brezhoneg",
        "value": "br"
      },
      {
        "name": "Català",
        "value": "ca"
      },
      {
        "name": "Eesti",
        "value": "et_EE"
      },
      {
        "name": "English (British English)",
        "value": "en_GB"
      },
      {
        "name": "Español (Argentina)",
        "value": "es_AR"
      },
      {
        "name": "Español (Ecuador)",
        "value": "es_EC"
      },
      {
        "name": "Español (México)",
        "value": "es_MX"
      },
      {
        "name": "Esperanto",
        "value": "eo"
      },
      {
        "name": "Euskara",
        "value": "eu"
      },
      {
        "name": "Galego",
        "value": "gl"
      },
      {
        "name": "Hrvatski",
        "value": "hr"
      },
      {
        "name": "Latviešu",
        "value": "lv"
      },
      {
        "name": "Lietuvių",
        "value": "lt_LT"
      },
      {
        "name": "Magyar",
        "value": "hu"
      },
      {
        "name": "Norsk bokmål",
        "value": "nb"
      },
      {
        "name": "Occitan",
        "value": "oc"
      },
      {
        "name": "Polski",
        "value": "pl"
      },
      {
        "name": "Română",
        "value": "ro"
      },
      {
        "name": "Slovenčina",
        "value": "sk"
      },
      {
        "name": "Slovenščina",
        "value": "sl"
      },
      {
        "name": "Tiếng Việt",
        "value": "vi"
      },
      {
        "name": "sardu",
        "value": "sc"
      },
      {
        "name": "suomi",
        "value": "fi"
      },
      {
        "name": "Íslenska",
        "value": "is"
      },
      {
        "name": "Čeština",
        "value": "cs"
      },
      {
        "name": "Ελληνικά",
        "value": "el"
      },
      {
        "name": "Български",
        "value": "bg"
      },
      {
        "name": "Македонски",
        "value": "mk"
      },
      {
        "name": "Српски",
        "value": "sr"
      },
      {
        "name": "Українська",
        "value": "uk"
      },
      {
        "name": "עברית",
        "value": "he"
      },
      {
        "name": "ئۇيغۇرچە",
        "value": "ug"
      },
      {
        "name": "فارسى",
        "value": "fa"
      },
      {
        "name": "ไทย",
        "value": "th"
      },
      {
        "name": "ຂີ້ຕົວະ",
        "value": "lo"
      },
      {
        "name": "ქართული ენა",
        "value": "ka"
      },
      {
        "name": "正體中文（臺灣）",
        "value": "zh_TW"
      },
      {
        "name": "正體中文（香港）",
        "value": "zh_HK"
      },
      {
        "name": "ga",
        "value": "ga"
      }
    ]
  ],
  "inputShapeDefaults": {
    "origin_language": "detect_language"
  },
  "inputShape": [
    {
      "name": "Origin text",
      "description": "The text to translate",
      "type": "Text"
    },
    {
      "name": "Origin language",
      "description": "The language of the origin text",
      "type": "Enum"
    },
    {
      "name": "Target language",
      "description": "The desired language to translate the origin text in",
      "type": "Enum"
    }
  ],
  "optionalInputShapeEnumValues": [
    []
  ],
  "optionalInputShapeDefaults": {
    "max_tokens": 1000,
    "model": "llama-3.3-70b-versatile"
  },
  "outputShape": [
    {
      "name": "Result",
      "description": "The translated text",
      "type": "Text"
    }
  ],
  "outputShapeEnumValues": [],
  "optionalOutputShape": [],
  "optionalOutputShapeEnumValues": []
}