Merge pull request #4 from innovatrics/feature/self-managed-s3

Feature/self managed s3
innovatrics · Nov 8, 2023 · b6101ba · b6101ba
2 parents ddfbc2a + cfdcee2
commit b6101ba
Show file tree

Hide file tree

Showing 9 changed files with 128 additions and 17 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,14 @@
 # Changelog
 
+## [v0.5.0]
+
+### Added
+- Support for using [minio subchart](https://github.com/bitnami/charts/tree/main/bitnami/minio) in place of S3 bucket
+  - This is now default behavior of the helm chart
+
+### Breaking change
+- MinIO subchart is enabled and used by default. To keep using S3 bucket managed outside of this helm chart please set the `minio.enabled` value to `false` and provide configuration details via `configurations.s3`
+
 ## [v0.4.0]
 
 ### Added

diff --git a/Chart.lock b/Chart.lock
@@ -5,5 +5,8 @@ dependencies:
 - name: rabbitmq
   repository: oci://registry-1.docker.io/bitnamicharts
   version: 12.0.4
-digest: sha256:15d5ba64f57411a0eebebdd610f019a86846d03d23278beb2c1765daf8137442
-generated: "2023-10-19T16:46:51.767222+02:00"
+- name: minio
+  repository: oci://registry-1.docker.io/bitnamicharts
+  version: 12.8.15
+digest: sha256:377f0abfb50bbc8665ab4176a9a24747d5bedc8604f4a867ab82006e8e3d2805
+generated: "2023-11-02T13:06:27.794117+01:00"
diff --git a/Chart.yaml b/Chart.yaml
@@ -11,6 +11,10 @@ dependencies:
     version: "12.0.4"
     repository: "oci://registry-1.docker.io/bitnamicharts"
     condition: rabbitmq.enabled
+  - name: minio
+    version: "12.8.15"
+    repository: "oci://registry-1.docker.io/bitnamicharts"
+    condition: minio.enabled
 
 type: application
 
@@ -28,6 +32,6 @@ maintainers:
 - name: Innovatrics
   url: https://www.innovatrics.com/
 
-version: 0.4.0
+version: 0.5.0
 
 appVersion: "v5_4.22.0"
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # smartface
 
-![Version: 0.4.0](https://img.shields.io/badge/Version-0.4.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v5_4.22.0](https://img.shields.io/badge/AppVersion-v5_4.22.0-informational?style=flat-square)
+![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v5_4.22.0](https://img.shields.io/badge/AppVersion-v5_4.22.0-informational?style=flat-square)
 
 SmartFace is a Scalable Facial Recognition Server Platform Able to Process Multiple Real-Time Video Streams. Currently the helm chart supports edge stream and Lightweight Face Identification System (LFIS) deployments
 
@@ -44,6 +44,7 @@ By default an ingress object is created with the helm chart. To configure the in
 | Repository | Name | Version |
 |------------|------|---------|
 | oci://ghcr.io/innovatrics/sf-helm | sf-tenant-operator | 0.2.0 |
+| oci://registry-1.docker.io/bitnamicharts | minio | 12.8.15 |
 | oci://registry-1.docker.io/bitnamicharts | rabbitmq | 12.0.4 |
 
 All chart dependencies are optional and can be disabled and supplemented with other (for example cloud-based) alternatives
@@ -82,6 +83,31 @@ stringData:
   rabbitmq-password: "<password>"
 ```
 
+### S3
+To use S3 bucket managed by AWS:
+- set `minio.enabled=false`
+- provide s3 configuration via:
+  - supplying values to `configurations.s3` object
+  - or creating ConfigMap and setting `configurations.s3.existingConfigMapName`
+  - see Sample objects for example
+- When using S3 bucket and running in AWS the authentication can be performed using could-native mechanisms:
+  - to authenticate using EC2 instance profile set `configurations.s3.authType` to `InstanceProfile`
+  - to authenticate using AssumedRole set `configurations.s3.authType` to `AssumedRole` (useful for example when using [EKS IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html))
+
+#### Sample objects
+```
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: "sf-s3-connection"
+data:
+  name: "smartface"
+  region: "eu-central-1"
+  folder: "sface"
+  authType: "AssumedRole"
+  useBucketRegion: "true"
+```
+
 ## Values
 
 | Key | Type | Default | Description |
@@ -409,6 +435,7 @@ stringData:
 | migration.initContainer.image.repository | string | `"innovatrics/smartface/sf-admin"` | Docker image repository |
 | migration.initContainer.image.tag | string | `nil` | Overrides the image tag whose default is the chart's appVersion |
 | migration.initContainer.resources | object | `{}` |  |
+| minio | object | `{"defaultBuckets":"smartface","enabled":true}` | config for minio subchart, see https://github.com/bitnami/charts/tree/main/bitnami/minio |
 | nameOverride | string | `nil` | Overrides the chart's name |
 | podAnnotations | object | `{}` | Common annotations for all pods |
 | podLabels | object | `{}` | Common labels for all pods |
@@ -499,6 +526,9 @@ stringData:
 
 ## Breaking changes
 
+### [v0.5.0]
+- MinIO subchart is enabled and used by default. To keep using S3 bucket managed outside of this helm chart please set the `minio.enabled` value to `false` and provide configuration details via `configurations.s3`
+
 ### [v0.4.0]
 - Changed default behavior for creating S3 configuration. If you like to continue managing the previously created S3 config map please use the `configurations.s3.existingConfigMapName` field. Otherwise the ConfigMap will be managed by the helm chart using the values provided in `configurations.s3`
   - The ConfigMap keys for existing config map are no longer configurable, so if you want to keep using the ConfigMap not managed by this chart then please make sure that the key match what the helm chart expects

diff --git a/README.md.gotmpl b/README.md.gotmpl
@@ -78,6 +78,31 @@ stringData:
   rabbitmq-password: "<password>"
 ```
 
+### S3
+To use S3 bucket managed by AWS:
+- set `minio.enabled=false`
+- provide s3 configuration via:
+  - supplying values to `configurations.s3` object
+  - or creating ConfigMap and setting `configurations.s3.existingConfigMapName`
+  - see Sample objects for example
+- When using S3 bucket and running in AWS the authentication can be performed using could-native mechanisms:
+  - to authenticate using EC2 instance profile set `configurations.s3.authType` to `InstanceProfile`
+  - to authenticate using AssumedRole set `configurations.s3.authType` to `AssumedRole` (useful for example when using [EKS IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html))
+
+#### Sample objects
+```
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: "sf-s3-connection"
+data:
+  name: "smartface"
+  region: "eu-central-1"
+  folder: "sface"
+  authType: "AssumedRole"
+  useBucketRegion: "true"
+```
+
 {{ template "chart.valuesSection" . }}
 
 {{ template "chart.maintainersSection" . }}
@@ -86,6 +111,9 @@ stringData:
 
 ## Breaking changes
 
+### [v0.5.0]
+- MinIO subchart is enabled and used by default. To keep using S3 bucket managed outside of this helm chart please set the `minio.enabled` value to `false` and provide configuration details via `configurations.s3`
+
 ### [v0.4.0]
 - Changed default behavior for creating S3 configuration. If you like to continue managing the previously created S3 config map please use the `configurations.s3.existingConfigMapName` field. Otherwise the ConfigMap will be managed by the helm chart using the values provided in `configurations.s3`
   - The ConfigMap keys for existing config map are no longer configurable, so if you want to keep using the ConfigMap not managed by this chart then please make sure that the key match what the helm chart expects

diff --git a/templates/_configurations.tpl b/templates/_configurations.tpl
@@ -26,23 +26,40 @@ Template used for adding S3 configuration to containers
     configMapKeyRef:
       name: {{ $configName | quote }}
       key: "region"
+      optional: true
 - name: "S3Bucket__Folder"
   valueFrom:
     configMapKeyRef:
       name: {{ $configName | quote }}
       key: "folder"
-# AssumedRole
+      optional: true
 - name: "S3Bucket__AuthenticationType"
   valueFrom:
     configMapKeyRef:
       name: {{ $configName | quote }}
       key: "authType"
-# BucketRegion
 - name: "S3Bucket__UseBucketRegion"
   valueFrom:
     configMapKeyRef:
       name: {{ $configName | quote }}
       key: "useBucketRegion"
+{{- if .Values.minio.enabled }}
+- name: "S3Bucket__Endpoint"
+  valueFrom:
+    configMapKeyRef:
+      name: {{ $configName | quote }}
+      key: "endpoint"
+- name: "S3Bucket__AccessKey"
+  valueFrom:
+    secretKeyRef:
+      name: "{{ .Release.Name }}-minio"
+      key: "root-user"
+- name: "S3Bucket__SecretKey"
+  valueFrom:
+    secretKeyRef:
+      name: "{{ .Release.Name }}-minio"
+      key: "root-password"
+{{- end }}
 {{- end }}
 
 {{/*

diff --git a/templates/_validations.tpl b/templates/_validations.tpl
@@ -5,20 +5,20 @@ Compile all warnings into a single message, and call fail.
 {{- define "smartface.validate" -}}
 {{- $messages := list -}}
 
-{{- $messages := append $messages (trim (include "smartface.validate.multitenantEdge" .)) -}}
-{{- $messages := append $messages (trim (include "smartface.validate.stationDeps" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.multitenantEdge" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.stationDeps" .)) -}}
 
 {{- if not .Values.skipLookupBasedValidations -}}
-{{- $messages := append $messages (trim (include "smartface.validate.dbConnectionSecret" .)) -}}
-{{- $messages := append $messages (trim (include "smartface.validate.s3Config" .)) -}}
-{{- $messages := append $messages (trim (include "smartface.validate.licenseSecret" .)) -}}
-{{- $messages := append $messages (trim (include "smartface.validate.authConfig" .)) -}}
-{{- $messages := append $messages (trim (include "smartface.validate.registryCreds" .)) -}}
-{{- $messages := append $messages (trim (include "smartface.validate.rmqConfig" .)) -}}
-{{- $messages := append $messages (trim (include "smartface.validate.mqttConfig" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.dbConnectionSecret" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.s3Config" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.licenseSecret" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.authConfig" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.registryCreds" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.rmqConfig" .)) -}}
+{{- $messages = append $messages (trim (include "smartface.validate.mqttConfig" .)) -}}
 {{- end -}}
 
-{{- $messages := without $messages "" -}}
+{{- $messages = without $messages "" -}}
 {{- $message := join "\n" $messages -}}
 {{- if $message -}}
 {{-   printf "\nVALIDATIONS:\n%s" $message | fail -}}
@@ -60,6 +60,11 @@ Validate that the S3 config map exists with correct keys
 */}}
 {{- define "smartface.validate.s3Config" -}}
 {{- $existingConfigMap := .Values.configurations.s3.existingConfigMapName -}}
+{{- if .Values.minio.enabled -}}
+{{- if $existingConfigMap }}
+Cannot deploy minio and use existing ConfigMap. Either disable minio deployment by setting `minio.enabled` to `false` or don't provide value for `configurations.s3.existingConfigMapName`
+{{- end }}
+{{- else}}
 {{- if $existingConfigMap -}}
 {{ include "smartface.validate.genericResourceWithKey" (dict "Version" "v1" "Type" "ConfigMap" "Namespace" .Release.Namespace "Name" $existingConfigMap "Key" "name") }}
 {{ include "smartface.validate.genericResourceWithKey" (dict "Version" "v1" "Type" "ConfigMap" "Namespace" .Release.Namespace "Name" $existingConfigMap "Key" "region") }}
@@ -75,6 +80,7 @@ Please provide value for `configurations.s3.bucketRegion`
 {{- end -}}
 {{- end -}}
 {{- end -}}
+{{- end -}}
 
 {{/*
 Validate that the license secret exists with correct keys

diff --git a/templates/s3-cm.yaml b/templates/s3-cm.yaml
@@ -1,12 +1,21 @@
-{{- if not .Values.configurations.s3.existingConfigMapName }}
+{{- if or .Values.minio.enabled (not .Values.configurations.s3.existingConfigMapName) }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: {{ include "smartface.s3.name" . | quote }}
 data:
+  {{- if .Values.minio.enabled }}
+  {{/* If we deploy minio, then values in cofig need to match the deployed minio, thus are not configurable */}}
+  name: "smartface"
+  endpoint: "http://{{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc.cluster.local:9000"
+  authType: "AccessKeyAndSecret"
+  useBucketRegion: "false"
+  {{- else }}
+  {{/* If the S3 bucket is provided externally, but user does not bring his own configmap, we need to use values */}}
   name: {{ .Values.configurations.s3.bucketName | quote }}
   region: {{ .Values.configurations.s3.bucketRegion | quote }}
   folder: {{ .Values.configurations.s3.bucketFolder | quote }}
   authType: {{ .Values.configurations.s3.authType | quote }}
   useBucketRegion: {{ .Values.configurations.s3.useBucketRegion | quote }}
+  {{- end }}
 {{- end }}
diff --git a/values.yaml b/values.yaml
@@ -117,6 +117,11 @@ rabbitmq:
       port: 5552
       targetPort: 5552
 
+# -- config for minio subchart, see https://github.com/bitnami/charts/tree/main/bitnami/minio
+minio:
+  enabled: true
+  defaultBuckets: "smartface"
+
 api:
   enabled: true
   name: "api"