Feature/embed quicksight (#233)

* Code to exchange entra token for aws token to test identity propogation

* Added commands to get local debugging working against apc dev. Added JWT as dependency

* Added quicksight embedding

* Add initial database access app

* Move quicksight template, use namespace url

* Use management account profile when running locally

* Initial aws service for Quicksight

* Add code to ensure single boto3 session created

* Add glueservice and more refactoring

* Refactor base AWS service

* Fix tests and linting

* Remove static analysis feature from devcontainer

Static analysis includes checkov, which installed a version of botocore
that conflicted with the version pinned in the requirements. It also
installed packages as the root user, which meant when trying to upgrade
botocore, a permission error was raised as the vscode user that runs the
dev container does not have permission to remove packages installed as
root. We dont need to run checkov locally so removing, which resolves
the issue.

* Add instructions to retreive .env file

* Attempt to appease superlinter

* Update the home and login failure pages

* Tidy up the frontend

* Prompt user to login when authenticating

* Update default behaviour when retreiving STS creds

We only want to default to using STS to retrieve AWS credentials when
not using a service role that already has permission to call services.
These changes attempt to make that more explicit. For now, we should
only default to using STS when running locally.

---------

Co-authored-by: jamesstottmoj <[email protected]>

.devcontainer/devcontainer-lock.json

@@ -32,11 +32,6 @@
       "version": "1.0.1",
       "resolved": "ghcr.io/ministryofjustice/devcontainer-feature/kubernetes@sha256:0ec758e44468ba2a8b70b87613762ab04e50f7bb5eac8f2aea592cff213dbde5",
       "integrity": "sha256:0ec758e44468ba2a8b70b87613762ab04e50f7bb5eac8f2aea592cff213dbde5"
-    },
-    "ghcr.io/ministryofjustice/devcontainer-feature/static-analysis:1": {
-      "version": "1.0.0",
-      "resolved": "ghcr.io/ministryofjustice/devcontainer-feature/static-analysis@sha256:e81d52725655c8ffb861605feac7ad155b447d51af65f6c3a03cab32d59f1e16",
-      "integrity": "sha256:e81d52725655c8ffb861605feac7ad155b447d51af65f6c3a03cab32d59f1e16"
     }
   }
-}
+}

.devcontainer/devcontainer.json

@@ -12,8 +12,7 @@
     "./features/src/postgresql": {},
     "ghcr.io/ministryofjustice/devcontainer-feature/aws:1": {},
     "ghcr.io/ministryofjustice/devcontainer-feature/container-structure-test:1": {},
-    "ghcr.io/ministryofjustice/devcontainer-feature/kubernetes:1": {},
-    "ghcr.io/ministryofjustice/devcontainer-feature/static-analysis:1": {}
+    "ghcr.io/ministryofjustice/devcontainer-feature/kubernetes:1": {}
   },
   "postCreateCommand": "bash .devcontainer/post-create.sh",
   "postStartCommand": "bash .devcontainer/post-start.sh",

.devcontainer/post-create.sh

@@ -18,3 +18,10 @@ make build-static
 
 # Run migrations
 python manage.py migrate
+
+# create aws and kube configs
+aws-sso config-profiles --force
+
+aws-sso exec --profile analytical-platform-development:AdministratorAccess -- aws eks --region eu-west-1 update-kubeconfig --name development-aWrhyc0m --alias dev-eks-cluster
+aws-sso exec --profile analytical-platform-compute-development:modernisation-platform-sandbox -- aws eks --region eu-west-2 update-kubeconfig --name analytical-platform-compute-development --alias apc-dev-cluster
+kubectl config use-context dev-eks-cluster

.github/workflows/super-linter.yml

@@ -42,3 +42,4 @@ jobs:
           PYTHON_ISORT_CONFIG_FILE: pyproject.toml
           PYTHON_MYPY_CONFIG_FILE: mypy.ini
           VALIDATE_KUBERNETES_KUBECONFORM: false # Super-Linter doesn't support https://github.com/jtyr/kubeconform-helm
+          VALIDATE_CHECKOV: false # TODO failures to remediate at later date

.gitignore

@@ -1,4 +1,5 @@
 .env
+.env.bak
 .terraform/
 coverage/
 venv/

Makefile

@@ -27,8 +27,9 @@ db-drop:
 serve:
 	python manage.py runserver
 
+# TODO revert this change
 serve-sso:
-	aws-sso exec --profile analytical-platform-compute-development:modernisation-platform-sandbox -- python manage.py runserver
+	aws-sso exec --profile analytical-platform-management-production:AdministratorAccess -- python manage.py runserver
 
 build-container:
 	@ARCH=`uname -m`; \

README.md

@@ -4,19 +4,27 @@
 
 ## Running Locally
 
-The dashboard is run in a DevContainer via Docker. The DevContainer VSCode extension is recommended, as is Docker Desktop.
+The dashboard is run in a DevContainer via Docker. The DevContainer Visual Studio Code extension is recommended, as is Docker Desktop.
 
 For more information on Dev Containers, see the [Analytical Platform docs.](https://technical-documentation.data-platform.service.justice.gov.uk/documentation/platform/infrastructure/developing.html#developing-the-data-platform)
 
 
 ### Building the DevContainer
-To build the dev container, ensure docker desktop is running, then open the AP UI project in VSCode. Open the command pallet by hitting command+shift+p and search for ```Dev Containers: Reopen in container``` and hit enter. This will build the dev container.
+To build the dev container, ensure docker desktop is running, then open the AP UI project in Visual Studio Code. Open the command pallet by hitting command+shift+p and search for ```Dev Containers: Reopen in container``` and hit enter. This will build the dev container.
 
 If you are using a workspace with multiple applications, search for ```Dev Containers: Open folder in Container…``` instead, then select the AP UI folder. Once the dev container has finished building, it should install all the required Python and npm dependencies, as well as run the migrations.
 
 ### Environment Variables
 There is an example environment file stored on 1Password named ```Analytical Platform UI Env```. Paste the contents into a new file called ```.env``` in the root of the project.
 
+If you have the 1password CLI installed on your local machine, you use the following command to copy the file:
+
+```bash
+op document get --vault "Analytical Platform" "Analytical Platform UI .env" --out-file .env
+```
+
+For installation instructions for the 1password CLI see [here](https://developer.1password.com/docs/cli/get-started/).
+
 ### Running Development Server
 To run the server, you will need to use aws-sso cli. To find the correct profile, run ```aws-sso list``` in the terminal. This will provide you with a link to sign in via SSO. Once signed in, a list of profiles will be displayed. You are looking for the profile name linked to the ```analytical-platform-compute-development``` AccountAlias.
 

ap/auth/oidc.py

@@ -1,6 +1,10 @@
 from django.conf import settings
 from django.utils import timezone
 
+import boto3
+import botocore
+import botocore.exceptions
+import jwt
 import structlog
 from authlib.integrations.django_client import OAuth
 
@@ -95,3 +99,48 @@ def expired(self):
         will be renewed based on t
         """
         return self._has_access_token_expired() or self._has_id_token_expired()
+
+
+# POC implementation
+def get_aws_identity_center_access_token(id_token):
+    """
+    Requires ID token for a user from EntraID
+    """
+    client = boto3.client("sso-oidc")
+    try:
+        response = client.create_token_with_iam(
+            clientId=settings.IDENTITY_CENTRE_OIDC_ARN,
+            grantType="urn:ietf:params:oauth:grant-type:jwt-bearer",
+            assertion=id_token,
+        )
+    except botocore.exceptions.ClientError as ice:
+        raise ice
+    except Exception as e:
+        raise e
+
+    # decode the ID token with pyjwt lib
+    aws_token = jwt.decode(jwt=response["idToken"], options={"verify_signature": False})
+    return aws_token
+
+
+def get_aws_credentials(aws_token):
+    """
+    Gets AWS credentials passing the identity context of the user from the AWS access token
+    """
+    sts = boto3.client("sts")
+    response = sts.assume_role(
+        RoleArn=settings.IAM_BEARER_ROLE_ARN,
+        RoleSessionName=f"identity-bearer-{aws_token['sub']}",
+        ProvidedContexts=[
+            {
+                "ProviderArn": "arn:aws:iam::aws:contextProvider/IdentityCenter",
+                "ContextAssertion": aws_token["sts:identity_context"],
+            },
+        ],
+    )
+    credentials = {
+        "aws_access_key_id": response["Credentials"]["AccessKeyId"],
+        "aws_secret_access_key": response["Credentials"]["SecretAccessKey"],
+        "aws_session_token": response["Credentials"]["SessionToken"],
+    }
+    return credentials

ap/auth/views/views.py

@@ -51,7 +51,7 @@ def _login_failure(self):
     def get(self, request):
         try:
             token = oauth.azure.authorize_access_token(request)
-            request.session["token"] = token
+            request.session["entra_access_token"] = token
             oidc_auth = OIDCSubAuthenticationBackend(token)
             user = oidc_auth.create_or_update_user()
             if not user:
@@ -60,6 +60,8 @@ def get(self, request):
                 self._login_success(request, user, token)
                 return redirect("/")
         except OAuthError as error:
+            if settings.DEBUG:
+                raise error
             sentry_sdk.capture_exception(error)
             return self._login_failure()
 
@@ -89,7 +91,7 @@ def get(self, request):
 class LoginFail(TemplateView):
     template_name = "login-fail.html"
 
-    def get_context_data(self, **kwargs):
-        context = super().get_context_data(**kwargs)
-        context["environment"] = settings.ENV
-        return context
+    def get(self, request, *args, **kwargs):
+        if self.request.user.is_authenticated:
+            return redirect("/")
+        return super().get(request, *args, **kwargs)

ap/aws/__init__.py

@@ -0,0 +1,5 @@
+from .base import AWSService
+from .glue import GlueService
+from .quicksight import QuicksightService
+
+__all__ = ["AWSService", "QuicksightService", "GlueService"]

ap/aws/base.py

@@ -0,0 +1,44 @@
+from django.conf import settings
+
+import boto3
+import botocore
+import sentry_sdk
+
+from . import session
+
+
+class AWSService:
+    aws_service_name: str = ""
+
+    def __init__(self, assume_role_name=None, profile_name=None, region_name=None):
+        self.assume_role_name = assume_role_name
+        self.profile_name = profile_name
+        self.region_name = region_name or settings.AWS_DEFAULT_REGION
+
+    @property
+    def credential_session_set(self) -> session.AWSCredentialSessionSet:
+        return session.AWSCredentialSessionSet()
+
+    @property
+    def boto3_session(self) -> boto3.Session:
+        return self.credential_session_set.get_or_create_session(
+            profile_name=self.profile_name,
+            assume_role_name=self.assume_role_name,
+            region_name=self.region_name,
+        )
+
+    @property
+    def client(self):
+        return self.boto3_session.client(self.aws_service_name)
+
+    def _request(self, method_name, **kwargs):
+        """
+        Make a request to the AWS service client. Handles exceptions and logs them to Sentry.
+        """
+        try:
+            return getattr(self.client, method_name)(**kwargs)
+        except botocore.exceptions.ClientError as e:
+            if settings.DEBUG:
+                raise e
+            sentry_sdk.capture_exception(e)
+            return None

ap/aws/glue.py

@@ -0,0 +1,31 @@
+from django.conf import settings
+
+from . import base
+
+
+class GlueService(base.AWSService):
+    aws_service_name = "glue"
+
+    def __init__(self, catalog_id=None):
+        super().__init__()
+        self.catalog_id = catalog_id or settings.GLUE_CATALOG_ID
+
+    def get_database_list(self):
+        databases = self._request("get_databases")
+        if not databases:
+            return []
+        return databases["DatabaseList"]
+
+    def get_table_list(self, database_name):
+        tables = self._request("get_tables", CatalogId=self.catalog_id, DatabaseName=database_name)
+        if not tables:
+            return []
+        return tables["TableList"]
+
+    def get_table_detail(self, database_name, table_name):
+        table = self._request(
+            "get_table", CatalogId=self.catalog_id, DatabaseName=database_name, Name=table_name
+        )
+        if not table:
+            return {}
+        return table["Table"]

ap/aws/quicksight.py

@@ -0,0 +1,25 @@
+from django.conf import settings
+
+from . import base
+
+
+class QuicksightService(base.AWSService):
+    aws_service_name = "quicksight"
+
+    def get_embed_url(self, user):
+        response = self._request(
+            "generate_embed_url_for_registered_user",
+            AwsAccountId=settings.COMPUTE_ACCOUNT_ID,
+            UserArn=user.quicksight_arn,
+            ExperienceConfiguration={
+                "QuickSightConsole": {
+                    "InitialPath": "/start",
+                    "FeatureConfigurations": {"StatePersistence": {"Enabled": True}},
+                },
+            },
+            AllowedDomains=settings.QUICKSIGHT_DOMAINS,
+        )
+        if response:
+            return response["EmbedUrl"]
+
+        return response

ap/aws/session.py

@@ -0,0 +1,102 @@
+from django.conf import settings
+
+import boto3
+import structlog
+from botocore import credentials
+from botocore.session import get_session
+
+log = structlog.getLogger(__name__)
+
+TTL = 1500
+
+
+class BotoSession:
+    def __init__(self, assume_role_name=None, profile_name=None, region_name=None):
+        self.assume_role_name = assume_role_name
+        if self.assume_role_name is None and settings.DEFAULT_STS_ROLE_TO_ASSUME is not None:
+            self.assume_role_name = settings.DEFAULT_STS_ROLE_TO_ASSUME
+
+        self.region_name = region_name or settings.AWS_DEFAULT_REGION
+        self.profile_name = profile_name
+
+    def refreshable_credentials(self):
+        log.info("Loading AWS credentials")
+        if not self.assume_role_name:
+            # boto3 refreshes the credentials automatically if no role is assumed
+            return self.get_default_credentials()
+
+        return credentials.RefreshableCredentials.create_from_metadata(
+            metadata=self.get_sts_credentials(),
+            refresh_using=self.get_sts_credentials,
+            method="sts-assume-role",
+        )
+
+    def get_sts_credentials(self) -> dict:
+        log.info("Getting credentials using STS")
+        boto3_ini_session = boto3.Session(region_name=settings.AWS_DEFAULT_REGION)
+        sts = boto3_ini_session.client("sts")
+        response = sts.assume_role(
+            RoleArn=self.assume_role_name,
+            RoleSessionName=f"analytical-platform-ui-{settings.ENV}",
+            DurationSeconds=TTL,
+        )
+        return {
+            "access_key": response["Credentials"]["AccessKeyId"],
+            "secret_key": response["Credentials"]["SecretAccessKey"],
+            "token": response["Credentials"]["SessionToken"],
+            "expiry_time": response["Credentials"]["Expiration"].isoformat(),
+        }
+
+    def get_default_credentials(self) -> credentials.Credentials:
+        log.info("Getting credentials using default boto3 method")
+        boto3_ini_session = boto3.Session(
+            region_name=self.region_name, profile_name=self.profile_name
+        )
+        return boto3_ini_session.get_credentials()
+
+    def get_boto3_session(self) -> boto3.Session:
+        log.info("Creating a new boto3 session")
+        botocore_session = get_session()
+        botocore_session.set_config_variable("region", self.region_name)
+        botocore_session._credentials = self.refreshable_credentials()
+        return boto3.Session(botocore_session=botocore_session)
+
+
+class SingletonMeta(type):
+    _instances: dict = {}
+
+    def __call__(cls, *args, **kwargs):
+        """
+        Possible changes to the value of the `__init__` argument do not affect
+        the returned instance.
+        """
+        if cls in cls._instances:
+            return cls._instances[cls]
+
+        instance = super().__call__(*args, **kwargs)
+        cls._instances[cls] = instance
+        return instance
+
+
+class AWSCredentialSessionSet(metaclass=SingletonMeta):
+    def __init__(self):
+        self.credential_sessions = {}
+
+    def get_or_create_session(
+        self,
+        profile_name: str | None = None,
+        assume_role_name: str | None = None,
+        region_name: str | None = None,
+    ) -> boto3.Session:
+        credential_session_key = "{}_{}_{}".format(profile_name, assume_role_name, region_name)
+        if credential_session_key in self.credential_sessions:
+            log.info(f"Returning existing session for {credential_session_key}")
+            return self.credential_sessions[credential_session_key]
+
+        log.warn(f"(for monitoring purpose) Initialising session ({credential_session_key})")
+        self.credential_sessions[credential_session_key] = BotoSession(
+            region_name=region_name,
+            profile_name=profile_name,
+            assume_role_name=assume_role_name,
+        ).get_boto3_session()
+        return self.credential_sessions[credential_session_key]