k8s-loki-s3

Grafana Loki is a horizontally scalable, highly available, multi-tenant log aggregation system which supports S3 as a storage backend.

Design

For detailed information, check out our Operator Guide for this bundle.

Usage

Our bundles aren't intended to be used locally, outside of testing. Instead, our bundles are designed to be configured, connected, deployed and monitored in the Massdriver platform.

What are Bundles?

Bundles are the basic building blocks of infrastructure, applications, and architectures in Massdriver. Read more here.

Bundle

Params

Form input parameters for configuring a bundle for deployment.

View

Properties

grafana (object)
- enabled (boolean): Enabling this will install Grafana for searching and viewing logs. If you have an existing Grafana instance you would prefer to use you can disable installation of this instance. Default: True.
loki (object)
- scalable (boolean): Loki can run in two modes: a simple, single binary "monolithic" mode, and a massively scalable, distributed mode. Monolithic consumes fewer resources and is suitable for smaller workloads (up to ~100GB per day). Scalable mode should be enabled for larger workloads, or if you would like to separate read and write concerns. Default: False.
namespace (string): Application will be deployed into this namespace. If the namespace doesn't exist, it will be created. Default: loki.
promtail (object)
- enabled (boolean): Promtail is the standard log aggregator and shipper for Loki. If you have an existing log shipper you prefer to use, you can disable installation of Promtail. Default: True.

Examples

{
    "__name": "Single Binary",
    "grafana": {
        "enabled": true
    },
    "loki": {
        "scalable": false,
        "singleBinary": {
            "replicas": 2
        }
    },
    "promtail": {
        "enabled": true
    }
}

{
    "__name": "Scalable",
    "grafana": {
        "enabled": true
    },
    "loki": {
        "backend": {
            "replicas": 3
        },
        "read": {
            "replicas": 3
        },
        "scalable": true,
        "write": {
            "replicas": 3
        }
    },
    "promtail": {
        "enabled": true
    }
}

Connections

Connections from other bundles that this bundle depends on.

View

Properties

aws_authentication (object): . Cannot contain additional properties.
- data (object)
  - arn (string): Amazon Resource Name.
    
    Examples:
```
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
```
```
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
```
  - external_id (string): An external ID is a piece of data that can be passed to the AssumeRole API of the Security Token Service (STS). You can then use the external ID in the condition element in a role's trust policy, allowing the role to be assumed only when a certain value is present in the external ID.
- specs (object)
  - aws (object): .
    - region (string): AWS Region to provision in.
      
      Examples:
      "us-west-2"
azure_authentication (object): . Cannot contain additional properties.
- data (object)
  - client_id (string): A valid UUID field.
    
    Examples:
```
"123xyz99-ab34-56cd-e7f8-456abc1q2w3e"
```
  - client_secret (string)
  - subscription_id (string): A valid UUID field.
    
    Examples:
```
"123xyz99-ab34-56cd-e7f8-456abc1q2w3e"
```
  - tenant_id (string): A valid UUID field.
    
    Examples:
```
"123xyz99-ab34-56cd-e7f8-456abc1q2w3e"
```
- specs (object)
bucket (object): Cannot contain additional properties.
- data (object)
  - infrastructure (object)
    - arn (string): Amazon Resource Name.
      
      Examples:
      "arn:aws:rds::ACCOUNT_NUMBER:db/prod"
      "arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
  - security (object): Informs downstream services of network and/or IAM policies. Cannot contain additional properties.
    - iam (object): IAM Policies. Cannot contain additional properties.
      - ^[a-z]+[a-z_]*[a-z]+$ (object)
        
        policy_arn (string): AWS IAM policy ARN.
        
        Examples:
        
        "arn:aws:rds::ACCOUNT_NUMBER:db/prod"
        
        "arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
    - identity (object): For instances where IAM policies must be attached to a role attached to an AWS resource, for instance AWS Eventbridge to Firehose, this attribute should be used to allow the downstream to attach it's policies (Firehose) directly to the IAM role created by the upstream (Eventbridge). It is important to remember that connections in massdriver are one way, this scheme perserves the dependency relationship while allowing bundles to control the lifecycles of resources under it's management. Cannot contain additional properties.
      - role_arn (string): ARN for this resources IAM Role.
        
        Examples:
        
        "arn:aws:rds::ACCOUNT_NUMBER:db/prod"
        
        "arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
    - network (object): AWS security group rules to inform downstream services of ports to open for communication. Cannot contain additional properties.
      - ^[a-z-]+$ (object)
        
        arn (string): Amazon Resource Name.
        
        Examples:
        
        "arn:aws:rds::ACCOUNT_NUMBER:db/prod"
        
        "arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
        
        port (integer): Port number. Minimum: 0. Maximum: 65535.
        
        protocol (string): Must be one of: ['tcp', 'udp'].
- specs (object)
  - aws (object): .
    - region (string): AWS Region to provision in.
      
      Examples:
      "us-west-2"
gcp_authentication (object): GCP Service Account. Cannot contain additional properties.
- data (object)
  - auth_provider_x509_cert_url (string): Auth Provider x509 Certificate URL. Default: https://www.googleapis.com/oauth2/v1/certs.
    
    Examples:
```
"https://example.com/some/path"
```
```
"https://massdriver.cloud"
```
  - auth_uri (string): Auth URI. Default: https://accounts.google.com/o/oauth2/auth.
    
    Examples:
```
"https://example.com/some/path"
```
```
"https://massdriver.cloud"
```
  - client_email (string): Service Account Email.
    
    Examples:
```
"[email protected]"
```
```
"[email protected]"
```
  - client_id (string): .
  - client_x509_cert_url (string): Client x509 Certificate URL.
    
    Examples:
```
"https://example.com/some/path"
```
```
"https://massdriver.cloud"
```
  - private_key (string): .
  - private_key_id (string): .
  - project_id (string): .
  - token_uri (string): Token URI. Default: https://oauth2.googleapis.com/token.
    
    Examples:
```
"https://example.com/some/path"
```
```
"https://massdriver.cloud"
```
  - type (string): . Default: service_account.
- specs (object)
  - gcp (object): .
    - project (string)
    - region (string): The GCP region to provision resources in.
      
      Examples:
      "us-east1"
      "us-east4"
      "us-west1"
      "us-west2"
      "us-west3"
      "us-west4"
      "us-central1"
kubernetes_cluster (object): Kubernetes cluster authentication and cloud-specific configuration. Cannot contain additional properties.
- data (object)
  - authentication (object)
    - cluster (object)
      - certificate-authority-data (string)
      - server (string)
    - user (object)
      - token (string)
  - infrastructure (object): Cloud specific Kubernetes configuration data.
    - One of
      - AWS EKS infrastructure configobject: . Cannot contain additional properties.
        
        arn (string): Amazon Resource Name.
        
        Examples:
        
        "arn:aws:rds::ACCOUNT_NUMBER:db/prod"
        
        "arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
        
        oidc_issuer_url (string): An HTTPS endpoint URL.
        
        Examples:
        
        "https://example.com/some/path"
        
        "https://massdriver.cloud"
      - Infrastructure Configobject: Azure AKS Infrastructure Configuration. Cannot contain additional properties.
        
        ari (string): Azure Resource ID.
        
        Examples:
        
        "/subscriptions/12345678-1234-1234-abcd-1234567890ab/resourceGroups/resource-group-name/providers/Microsoft.Network/virtualNetworks/network-name"
        
        oidc_issuer_url (string)
      - GCP Infrastructure GRNobject: Minimal GCP Infrastructure Config. Cannot contain additional properties.
        
        grn (string): GCP Resource Name (GRN).
        
        Examples:
        
        "projects/my-project/global/networks/my-global-network"
        
        "projects/my-project/regions/us-west2/subnetworks/my-subnetwork"
        
        "projects/my-project/topics/my-pubsub-topic"
        
        "projects/my-project/subscriptions/my-pubsub-subscription"
        
        "projects/my-project/locations/us-west2/instances/my-redis-instance"
        
        "projects/my-project/locations/us-west2/clusters/my-gke-cluster"
- specs (object)
  - aws (object): .
    - region (string): AWS Region to provision in.
      
      Examples:
      "us-west-2"
  - azure (object): .
    - region (string): Select the Azure region you'd like to provision your resources in.
  - gcp (object): .
    - project (string)
    - region (string): The GCP region to provision resources in.
      
      Examples:
      "us-east1"
      "us-east4"
      "us-west1"
      "us-west2"
      "us-west3"
      "us-west4"
      "us-central1"
  - kubernetes (object): Kubernetes distribution and version specifications.
    - cloud (string): Must be one of: ['aws', 'gcp', 'azure'].
    - distribution (string): Must be one of: ['eks', 'gke', 'aks'].
    - platform_version (string)
    - version (string)

Artifacts

Resources created by this bundle that can be connected to other bundles.

View

Properties

Contributing

Bug Reports & Feature Requests

Did we miss something? Please submit an issue to report any bugs or request additional features.

Developing

Note: Massdriver bundles are intended to be tightly use-case scoped, intention-based, reusable pieces of IaC for use in the Massdriver platform. For this reason, major feature additions that broaden the scope of an existing bundle are likely to be rejected by the community.

Still want to get involved? First check out our contribution guidelines.

Fix or Fork

If your use-case isn't covered by this bundle, you can still get involved! Massdriver is designed to be an extensible platform. Fork this bundle, or create your own bundle from scratch!

Connect

Questions? Concerns? Adulations? We'd love to hear from you!

Please connect with us!

Name		Name	Last commit message	Last commit date
Latest commit History 17 Commits
.github/workflows		.github/workflows
src		src
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
README.md		README.md
icon.svg		icon.svg
massdriver.yaml		massdriver.yaml
operator.md		operator.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

k8s-loki-s3

Design