-
Notifications
You must be signed in to change notification settings - Fork 37
/
metadata.yaml
144 lines (143 loc) · 5.73 KB
/
metadata.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: blueprints.cloud.google.com/v1alpha1
kind: BlueprintMetadata
metadata:
name: terraform-genai-knowledge-base
annotations:
config.kubernetes.io/local-config: "true"
spec:
info:
title: Generative AI Knowledge Base
source:
repo: https://github.com/GoogleCloudPlatform/terraform-genai-knowledge-base.git
sourceType: git
version: 0.3.0
actuationTool:
flavor: Terraform
version: ">= 0.13"
description:
tagline: Fine tune an LLM model to answer questions from your documents.
detailed: This solution showcases how to extract question & answer pairs out of documents using Generative AI. It provides an end-to-end demonstration of QA extraction and fine-tuning of a large language model (LLM) on Vertex AI. Along the way, the solution utilizes Document AI Character Recognition (OCR), Firestore, Vector Search, Vertex AI Studio, and Cloud Functions.
icon: assets/icon.png
deploymentDuration:
configurationSecs: "120"
deploymentSecs: "360"
costEstimate:
description: Cost Details
url: https://cloud.google.com/products/calculator-legacy#id=dcdd11ca-ca20-428b-86be-1c9b3c651042
cloudProducts:
- productId: ai-platform
label: Vertex AI
- productId: document-ai
label: Document AI OCR
- productId: FIRESTORE_SECTION
label: Firestore
- productId: FUNCTIONS_SECTION
label: Cloud Functions
- productId: STORAGE_SECTION
label: Cloud Storage
- productId: EVENTARC_SECTION
label: Eventarc
content:
architecture:
diagramUrl: assets/architecture_diagram.svg
description:
- Uploading a new document triggers the webhook Cloud Function.
- Document AI extracts the text from the document file.
- Indexes the document text in Vector Search.
- A Vertex AI Large Language Model generates questions and answers from the document text.
- The questions and answers pairs are saved into Firestore.
- A fine tuning dataset is generated from the Firestore database.
- After human validation, a fine tuned Large Language Model is deployed and saved in the Model Registry.
documentation:
- title: Knowledge Base using Generative AI
url: assets/architecture_diagram.svg
examples:
- name: simple_example
location: examples/simple_example
interfaces:
variables:
- name: disable_services_on_destroy
description: Whether project services will be disabled when the resources are destroyed.
varType: bool
defaultValue: false
- name: documentai_location
description: Document AI location, see https://cloud.google.com/document-ai/docs/regions
varType: string
defaultValue: us
- name: firestore_location
description: Firestore location, see https://firebase.google.com/docs/firestore/locations
varType: string
defaultValue: nam5
- name: labels
description: A set of key/value label pairs to assign to the resources deployed by this blueprint.
varType: map(string)
defaultValue: {}
- name: project_id
description: The Google Cloud project ID to deploy to
varType: string
defaultValue: null
required: true
- name: region
description: The Google Cloud region to deploy to
varType: string
defaultValue: us-central1
- name: unique_names
description: Whether to use unique names for resources
varType: bool
defaultValue: false
outputs:
- name: bucket_docs_name
description: The name of the docs bucket created
- name: bucket_main_name
description: The name of the main bucket created
- name: docs_index_endpoint_id
description: The ID of the docs index endpoint
- name: docs_index_id
description: The ID of the docs index
- name: documentai_processor_id
description: The full Document AI processor path ID
- name: firestore_database_name
description: The name of the Firestore database created
- name: neos_tutorial_url
description: The URL to launch the in-console tutorial for the Generative AI Knowledge Base solution
- name: predictions_notebook_url
description: The URL to open the notebook for model predictions in Colab
- name: unique_id
description: The unique ID for this deployment
requirements:
roles:
- level: Project
roles:
- roles/aiplatform.user
- roles/artifactregistry.admin
- roles/documentai.editor
- roles/firebase.admin
- roles/iam.serviceAccountAdmin
- roles/iam.serviceAccountUser
- roles/resourcemanager.projectIamAdmin
- roles/serviceusage.serviceUsageAdmin
services:
- aiplatform.googleapis.com
- artifactregistry.googleapis.com
- cloudbuild.googleapis.com
- cloudfunctions.googleapis.com
- cloudresourcemanager.googleapis.com
- documentai.googleapis.com
- eventarc.googleapis.com
- firestore.googleapis.com
- run.googleapis.com
- serviceusage.googleapis.com
- storage.googleapis.com