diff --git a/README.md b/README.md index dbbaeefe..a0fa69b2 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,14 @@ [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black) -> A [Django](https://docs.djangoproject.com) project template ready for continuous delivery. +> A [Django](https://docs.djangoproject.com) service template aligned to the 20tab **Minos** platform model: per-env Vault-driven secrets, GitLab Components OpenTofu deploys, Terraform Cloud workspaces managed by the parent platform. + +The generated service is meant to live as a sibling sub-repo of a platform produced by [talos](https://github.com/20tab/talos), and ships with: + +- `Dockerfile` multi-stage on `uv` + Python 3.14 +- `.gitlab-ci.yml` using `${CI_SERVER_FQDN}/components/opentofu/apply` and `registry.gitlab.com/20tab-open/minos/service:latest` +- `minos/{development,staging,production}/this.tfvars` + `common.tfvars` per-env configs +- Vault secret consumption at `{project}/envs/${CI_ENVIRONMENT_SLUG}/{service}/...` ## 🧩 Requirements @@ -33,15 +40,20 @@ python3 -m pip install -r requirements/common.txt The `terraform` cli package is required, unless you want to generate a project only locally. To install it we suggest to use the official [install guide](https://learn.hashicorp.com/tutorials/terraform/install-cli). -## πŸ”‘ Credentials (optional) +## πŸ”‘ Prerequisites -### 🦊 GitLab +### πŸ—οΈ Vault project (one-time, admin) + +The Minos pipeline assumes a shared Vault auth backbone is already provisioned by the [vault-project](https://github.com/20tab/vault-project) admin repo: KV mount, GitLab JWT auth backend, JWT roles `service-gitlab-job` and `platform-gitlab-job`, identity entity, admin policy. Run that **once per Vault cluster, before** bootstrapping any platform/service. + +This sub-bootstrapper only seeds **service-scoped** secrets at `{project_slug}/envs/{env}/{service_slug}/...`. Vault prompts are optional: skip them if Vault is not used for this project. -If the GitLab integration is enabled, a Personal Access Token with _api_ permission is required.
-It can be generated in the GitLab User Settings panel. +### 🦊 GitLab (optional) -**Note:** the token can be generated in the Access Tokens section of the GitLab User Settings panel.
-⚠️ Beware that the token is shown only once after creation. +If the GitLab integration is enabled, a Personal Access Token with _api_ scope is required.
+It can be generated in the GitLab User Settings β†’ Access Tokens panel. + +⚠️ The token is shown only once after creation. ## πŸš€οΈ Quickstart @@ -70,41 +82,34 @@ source talos-django/.venv/bin/activate Project name: My Project Name Project slug [my-project-name]: Service slug [backend]: -Project dirname (backend, myprojectname) [backend]: myprojectname -Deploy type (digitalocean-k8s, other-k8s) [digitalocean-k8s]: -Terraform backend (gitlab, terraform-cloud) [terraform-cloud]: -Terraform host name [app.terraform.io]: -Terraform Cloud User token: -Terraform Organization: my-organization-name -Do you want to create Terraform Cloud Organization 'my-organization-name'? [y/N]: -Choose the environments distribution: - 1 - All environments share the same stack (Default) - 2 - Dev and Stage environments share the same stack, Prod has its own - 3 - Each environment has its own stack - (1, 2, 3) [1]: +Project dirname (backend, myprojectname) [backend]: +Do you want to use Redis? [y/N]: +Do you want to use Postgres? [Y/n]: +Create a database inside the Postgres cluster? [Y/n]: +Terraform Cloud organization: my-tfc-org +Do you want to use Vault for secrets management? [y/N]: y +Vault token (leave blank to perform a browser-based OIDC authentication): +Make sure your Vault permissions allow to enable the project secrets backends and manage the project secrets. Continue? [y/N]: y +Vault address: https://vault.example.com +Cluster slug hosting the 'development' environment [dev]: +Cluster slug hosting the 'staging' environment [dev]: +Cluster slug hosting the 'production' environment [main]: Development environment complete URL [https://dev.my-project-name.com]: Staging environment complete URL [https://stage.my-project-name.com]: Production environment complete URL [https://www.my-project-name.com]: -Media storage (digitalocean-s3, aws-s3, local, none) [digitalocean-s3]: -Do you want to configure Redis? [y/N]: +Do you want to use Sentry? [y/N]: Do you want to use GitLab? [Y/n]: -GitLab group slug [my-project-name]: -Make sure the GitLab "my-project-name" group exists before proceeding. Continue? [y/N]: y -GitLab private token (with API scope enabled): -Sentry DSN (leave blank if unused) []: +GitLab URL [https://gitlab.com]: +GitLab access token (with API scope enabled): +GitLab parent group path: 20tab/my-project-name +Media storage (digitalocean-s3, aws-s3, local, none) [digitalocean-s3]: Initializing the backend service: ...cookiecutting the service ...generating the .env file ...formatting the cookiecut python code -...compiling the requirements files - - common.txt - - test.txt - - local.txt - - remote.txt - - base.txt ...creating the '/static' directory ...creating the GitLab repository and associated resources -...creating the Terraform Cloud resources +...creating the Vault resources with Terraform ``` ## πŸ—’οΈ Arguments @@ -147,43 +152,22 @@ The following arguments can be appended to the Docker and shell commands ### πŸ“ Architecture -#### Deploy type +#### Terraform Cloud organization -| Description | Argument | -| ----------------------- | ------------------------------------ | -| DigitalOcean Kubernetes | `--deployment-type=digitalocean-k8s` | -| Other Kubernetes | `--deployment-type=other-k8s` | +The TFC organization that owns the service workspaces. The workspaces themselves (`{project}_{service}_{env}`) are created by the parent platform via [talos](https://github.com/20tab/talos), not here. -#### Terraform backend +`--terraform-cloud-organization=my-tfc-org` -| Name | Argument | -| --------------- | ------------------------------------- | -| Terraform Cloud | `--terraform-backend=terraform-cloud` | -| GitLab | `--terraform-backend=gitlab` | +#### Cluster mapping per environment -##### Terraform Cloud required argument +Each environment is deployed to one cluster. Cluster slugs are prompted interactively per env (defaults: `development β†’ dev`, `staging β†’ dev`, `production β†’ main`). There is no CLI flag for this mapping; pass them via prompt or `--quiet` with the defaults. -`--terraform-cloud-hostname=app.terraform.io`
-`--terraform-cloud-token={{terraform-cloud-token}}`
-`--terraform-cloud-organization` +#### πŸ—οΈ Vault -##### Terraform Cloud create organization +`--vault-url=https://vault.example.com`
+`--vault-token={{vault-token}}` (env var: `VAULT_TOKEN`; leave blank for browser-based OIDC) -`--terraform-cloud-organization-create`
-`--terraform-cloud-admin-email={{terraform-cloud-admin-email}}` - -Disabled args -`--terraform-cloud-organization-create-skip` - -#### Environment distribution - -Choose the environments distribution: - -| Value | Description | Argument | -| ----- | ----------------------------------------------------------------- | ------------------------------ | -| 1 | All environments share the same stack (Default) | `--environment-distribution=1` | -| 2 | Dev and Stage environments share the same stack, Prod has its own | `--environment-distribution=2` | -| 3 | Each environment has its own stack | `--environment-distribution=3` | +Omit `--vault-url` to disable Vault integration (in that case GitLab CI vars are used as a fallback for sensitive values). #### Project Domain @@ -213,21 +197,37 @@ Disabled args ### 🦊 GitLab -> **⚠️ Important: Make sure the GitLab group exists before creating.** > https://gitlab.com/gitlab-org/gitlab/-/issues/244345 - For enabling gitlab integration the following arguments are needed: -`--gitlab-private-token={{gitlab-private-token}}`
-`--gitlab-group-path={{gitlab-group-path}}` +`--gitlab-url=https://gitlab.com`
+`--gitlab-token={{gitlab-token}}` (env var: `GITLAB_PRIVATE_TOKEN`)
+`--gitlab-namespace-path=20tab/my-project-name` + +The namespace path can be nested (e.g. `20tab/my-project-name`). When invoked from talos, this is set automatically to `{parent-group}/{project-slug}`. #### πŸͺ– Sentry For enabling sentry integration the following arguments are needed: -`--sentry-dsn={{frontend-sentry-dsn}}` +`--sentry-org={{sentry-org}}`
+`--sentry-url=https://sentry.io/`
+`--sentry-dsn={{sentry-dsn}}` #### πŸ”‡ Quiet No confirmations shown. `--quiet` + +### 🧰 Toolchain version overrides + +The generated service pins specific versions of Python, OpenTofu and the Minos image. Defaults match the current 20tab platform; override only if needed. + +| Field | Default | Where it lands | +| ---------------------------- | -------------------------------------------------------- | ------------------------------------------- | +| `python_version` | `3.14` | `Dockerfile`, `pyproject.toml` (ruff/mypy) | +| `minos_service_image` | `registry.gitlab.com/20tab-open/minos/service:latest` | `.gitlab-ci.yml` deploy image | +| `opentofu_component_version` | `3.11.0` | GitLab Component pin in `.gitlab-ci.yml` | +| `opentofu_version` | `1.10.6` | OpenTofu binary version in `.gitlab-ci.yml` | + +These are not exposed as CLI flags; pass them as kwargs when invoking the `Runner` directly (e.g. from talos). diff --git a/bootstrap/collector.py b/bootstrap/collector.py index 287a37e1..945a2dc7 100644 --- a/bootstrap/collector.py +++ b/bootstrap/collector.py @@ -9,15 +9,15 @@ from slugify import slugify from bootstrap.constants import ( - DEPLOYMENT_TYPE_CHOICES, - DEPLOYMENT_TYPE_DIGITALOCEAN, - DEPLOYMENT_TYPE_OTHER, - ENVIRONMENTS_DISTRIBUTION_CHOICES, - ENVIRONMENTS_DISTRIBUTION_DEFAULT, - ENVIRONMENTS_DISTRIBUTION_PROMPT, + ENV_NAMES, + ENV_TO_CLUSTER_DEFAULT, GITLAB_URL_DEFAULT, MEDIA_STORAGE_CHOICES, MEDIA_STORAGE_DIGITALOCEAN_S3, + MINOS_SERVICE_IMAGE, + OPENTOFU_COMPONENT_VERSION, + OPENTOFU_VERSION, + PYTHON_VERSION_DEFAULT, TERRAFORM_BACKEND_CHOICES, TERRAFORM_BACKEND_TFC, ) @@ -43,7 +43,6 @@ class Collector: project_dirname: str | None = None service_slug: str | None = None internal_service_port: int | None = None - deployment_type: str | None = None terraform_backend: str | None = None terraform_cloud_hostname: str | None = None terraform_cloud_token: str | None = None @@ -52,7 +51,9 @@ class Collector: terraform_cloud_admin_email: str | None = None vault_token: str | None = None vault_url: str | None = None - environments_distribution: str | None = None + use_postgres: bool | None = None + postgres_create_database: bool | None = None + env_to_cluster: dict[str, str] | None = None project_url_dev: str | None = None project_url_stage: str | None = None project_url_prod: str | None = None @@ -64,6 +65,10 @@ class Collector: gitlab_url: str | None = None gitlab_token: str | None = None gitlab_namespace_path: str | None = None + python_version: str | None = None + minos_service_image: str | None = None + opentofu_component_version: str | None = None + opentofu_version: str | None = None uid: int | None = None gid: int | None = None terraform_dir: Path | None = None @@ -81,14 +86,15 @@ def collect(self): self.set_project_dirname() self.set_service_dir() self.set_use_redis() + self.set_postgres() self.set_terraform() self.set_vault() - self.set_deployment_type() - self.set_environments_distribution() + self.set_env_to_cluster() self.set_project_urls() self.set_sentry() self.set_gitlab() self.set_media_storage() + self.set_versions() def set_project_slug(self): """Set the project slug option.""" @@ -133,6 +139,18 @@ def set_use_redis(self): warning("Do you want to use Redis?"), default=False ) + def set_postgres(self): + """Set the Postgres options.""" + if self.use_postgres is None: + self.use_postgres = click.confirm( + warning("Do you want to use Postgres?"), default=True + ) + if self.use_postgres and self.postgres_create_database is None: + self.postgres_create_database = click.confirm( + warning("Create a database inside the Postgres cluster?"), + default=True, + ) + def set_terraform(self): """Set the Terraform options.""" if self.terraform_backend not in TERRAFORM_BACKEND_CHOICES: @@ -192,27 +210,15 @@ def set_vault(self): ) self.vault_url = validate_or_prompt_url("Vault address", self.vault_url) - def set_deployment_type(self): - """Set the deployment type option.""" - if self.deployment_type not in DEPLOYMENT_TYPE_CHOICES: - self.deployment_type = click.prompt( - "Deploy type", - default=DEPLOYMENT_TYPE_DIGITALOCEAN, - type=click.Choice(DEPLOYMENT_TYPE_CHOICES, case_sensitive=False), - ).lower() - - def set_environments_distribution(self): - """Set the environments distribution option.""" - # TODO: forcing a single stack when deployment is `k8s-other` should be removed, - # and `set_deployment_type` merged with `set_deployment` - if self.deployment_type == DEPLOYMENT_TYPE_OTHER: - self.environments_distribution = "1" - elif self.environments_distribution not in ENVIRONMENTS_DISTRIBUTION_CHOICES: - self.environments_distribution = click.prompt( - ENVIRONMENTS_DISTRIBUTION_PROMPT, - default=ENVIRONMENTS_DISTRIBUTION_DEFAULT, - type=click.Choice(ENVIRONMENTS_DISTRIBUTION_CHOICES), - ) + def set_env_to_cluster(self): + """Set the environment-to-cluster mapping (one cluster slug per environment).""" + self.env_to_cluster = self.env_to_cluster or {} + for env_name in ENV_NAMES: + if env_name not in self.env_to_cluster: + self.env_to_cluster[env_name] = click.prompt( + f"Cluster slug hosting the '{env_name}' environment", + default=ENV_TO_CLUSTER_DEFAULT[env_name], + ) def set_project_urls(self): """Set the project urls options.""" @@ -286,6 +292,21 @@ def set_media_storage(self): type=click.Choice(MEDIA_STORAGE_CHOICES, case_sensitive=False), ).lower() + def set_versions(self): + """Set the toolchain versions.""" + self.python_version = self.python_version or click.prompt( + "Python version", default=PYTHON_VERSION_DEFAULT + ) + self.minos_service_image = self.minos_service_image or click.prompt( + "Minos service image", default=MINOS_SERVICE_IMAGE + ) + self.opentofu_component_version = self.opentofu_component_version or click.prompt( + "OpenTofu CI component version", default=OPENTOFU_COMPONENT_VERSION + ) + self.opentofu_version = self.opentofu_version or click.prompt( + "OpenTofu version", default=OPENTOFU_VERSION + ) + def get_runner(self): """Get the bootstrap runner instance.""" return Runner( @@ -298,7 +319,6 @@ def get_runner(self): service_dir=self._service_dir, service_slug=self.service_slug, internal_service_port=self.internal_service_port, - deployment_type=self.deployment_type, terraform_backend=self.terraform_backend, terraform_cloud_hostname=self.terraform_cloud_hostname, terraform_cloud_token=self.terraform_cloud_token, @@ -307,7 +327,9 @@ def get_runner(self): terraform_cloud_admin_email=self.terraform_cloud_admin_email, vault_token=self.vault_token, vault_url=self.vault_url, - environments_distribution=self.environments_distribution, + use_postgres=self.use_postgres, + postgres_create_database=self.postgres_create_database, + env_to_cluster=self.env_to_cluster, project_url_dev=self.project_url_dev, project_url_stage=self.project_url_stage, project_url_prod=self.project_url_prod, @@ -319,6 +341,10 @@ def get_runner(self): gitlab_url=self.gitlab_url, gitlab_token=self.gitlab_token, gitlab_namespace_path=self.gitlab_namespace_path, + python_version=self.python_version, + minos_service_image=self.minos_service_image, + opentofu_component_version=self.opentofu_component_version, + opentofu_version=self.opentofu_version, terraform_dir=self.terraform_dir, logs_dir=self.logs_dir, ) diff --git a/bootstrap/constants.py b/bootstrap/constants.py index 2759ce74..6e5aa45d 100644 --- a/bootstrap/constants.py +++ b/bootstrap/constants.py @@ -1,34 +1,5 @@ """Web project initialization CLI constants.""" -# Stacks - -# BEWARE: stack names must be suitable for inclusion in Vault paths - -DEV_STACK_NAME = "development" - -DEV_STACK_SLUG = "dev" - -STAGE_STACK_NAME = "staging" - -STAGE_STACK_SLUG = "stage" - -MAIN_STACK_NAME = "main" - -MAIN_STACK_SLUG = "main" - -STACKS_CHOICES = { - "1": [{"name": MAIN_STACK_NAME, "slug": MAIN_STACK_SLUG}], - "2": [ - {"name": DEV_STACK_NAME, "slug": DEV_STACK_SLUG}, - {"name": MAIN_STACK_NAME, "slug": MAIN_STACK_SLUG}, - ], - "3": [ - {"name": DEV_STACK_NAME, "slug": DEV_STACK_SLUG}, - {"name": STAGE_STACK_NAME, "slug": STAGE_STACK_SLUG}, - {"name": MAIN_STACK_NAME, "slug": MAIN_STACK_SLUG}, - ], -} - # Environments # BEWARE: environment names must be suitable for inclusion in Vault paths @@ -37,24 +8,15 @@ DEV_ENV_SLUG = "dev" -DEV_ENV_STACK_CHOICES: dict[str, str] = { - "1": MAIN_STACK_SLUG, -} - STAGE_ENV_NAME = "staging" STAGE_ENV_SLUG = "stage" -STAGE_ENV_STACK_CHOICES: dict[str, str] = { - "1": MAIN_STACK_SLUG, - "2": DEV_STACK_SLUG, -} - PROD_ENV_NAME = "production" PROD_ENV_SLUG = "prod" -PROD_ENV_STACK_CHOICES: dict[str, str] = {} +ENV_NAMES = [DEV_ENV_NAME, STAGE_ENV_NAME, PROD_ENV_NAME] # Env vars @@ -62,26 +24,6 @@ VAULT_TOKEN_ENV_VAR = "VAULT_TOKEN" # nosec B105 -# Deployment type - -DEPLOYMENT_TYPE_DIGITALOCEAN = "digitalocean-k8s" - -DEPLOYMENT_TYPE_OTHER = "other-k8s" - -DEPLOYMENT_TYPE_CHOICES = [DEPLOYMENT_TYPE_DIGITALOCEAN, DEPLOYMENT_TYPE_OTHER] - -# Environments distribution - -ENVIRONMENTS_DISTRIBUTION_DEFAULT = "1" - -ENVIRONMENTS_DISTRIBUTION_CHOICES = [ENVIRONMENTS_DISTRIBUTION_DEFAULT, "2", "3"] - -ENVIRONMENTS_DISTRIBUTION_PROMPT = """Choose the environments distribution: - 1 - All environments share the same stack (Default) - 2 - Dev and Stage environments share the same stack, Prod has its own - 3 - Each environment has its own stack -""" - # Media storage MEDIA_STORAGE_DIGITALOCEAN_S3 = "digitalocean-s3" @@ -95,7 +37,6 @@ "none", ] - # Terraform backend TERRAFORM_BACKEND_GITLAB = "gitlab" @@ -107,3 +48,33 @@ # GitLab GITLAB_URL_DEFAULT = "https://gitlab.com" + +# Clusters + +CLUSTER_DEV_SLUG = "dev" + +CLUSTER_MAIN_SLUG = "main" + +ENV_TO_CLUSTER_DEFAULT: dict[str, str] = { + DEV_ENV_NAME: CLUSTER_DEV_SLUG, + STAGE_ENV_NAME: CLUSTER_DEV_SLUG, + PROD_ENV_NAME: CLUSTER_MAIN_SLUG, +} + +# Vault + +VAULT_SERVICE_ROLE = "service-gitlab-job" + +# Minos + +MINOS_SERVICE_IMAGE = "registry.gitlab.com/20tab-open/minos/service:latest" + +# OpenTofu + +OPENTOFU_COMPONENT_VERSION = "3.11.0" + +OPENTOFU_VERSION = "1.10.6" + +# Python + +PYTHON_VERSION_DEFAULT = "3.14" diff --git a/bootstrap/runner.py b/bootstrap/runner.py index 96ea1ad7..f3ebfcd3 100644 --- a/bootstrap/runner.py +++ b/bootstrap/runner.py @@ -17,18 +17,15 @@ from bootstrap.constants import ( DEV_ENV_NAME, DEV_ENV_SLUG, - DEV_ENV_STACK_CHOICES, - DEV_STACK_SLUG, GITLAB_URL_DEFAULT, - MAIN_STACK_SLUG, + MINOS_SERVICE_IMAGE, + OPENTOFU_COMPONENT_VERSION, + OPENTOFU_VERSION, PROD_ENV_NAME, PROD_ENV_SLUG, - PROD_ENV_STACK_CHOICES, - STACKS_CHOICES, + PYTHON_VERSION_DEFAULT, STAGE_ENV_NAME, STAGE_ENV_SLUG, - STAGE_ENV_STACK_CHOICES, - STAGE_STACK_SLUG, TERRAFORM_BACKEND_TFC, ) from bootstrap.exceptions import BootstrapError @@ -55,8 +52,7 @@ class Runner: service_dir: Path service_slug: str internal_service_port: int - deployment_type: str - environments_distribution: str + env_to_cluster: dict[str, str] project_url_dev: str = "" project_url_stage: str = "" project_url_prod: str = "" @@ -65,6 +61,7 @@ class Runner: terraform_cloud_token: str | None = None terraform_cloud_organization: str | None = None terraform_cloud_organization_create: bool | None = None + terraform_cloud_project_create: bool = True terraform_cloud_admin_email: str | None = None vault_token: str | None = None vault_url: str | None = None @@ -73,15 +70,20 @@ class Runner: sentry_url: str | None = None media_storage: str use_redis: bool = False + use_postgres: bool = True + postgres_create_database: bool = True gitlab_url: str | None = None gitlab_namespace_path: str | None = None gitlab_token: str | None = None + python_version: str = PYTHON_VERSION_DEFAULT + minos_service_image: str = MINOS_SERVICE_IMAGE + opentofu_component_version: str = OPENTOFU_COMPONENT_VERSION + opentofu_version: str = OPENTOFU_VERSION uid: int | None = None gid: int | None = None terraform_dir: Path | None = None logs_dir: Path | None = None run_id: str = field(init=False) - stacks: list = field(init=False, default_factory=list) envs: list = field(init=False, default_factory=list) gitlab_variables: dict = field(init=False, default_factory=dict) tfvars: dict = field(init=False, default_factory=dict) @@ -95,45 +97,27 @@ def __post_init__(self): self.run_id = f"{time():.0f}" self.terraform_dir = self.terraform_dir or Path(f".terraform/{self.run_id}") self.logs_dir = self.logs_dir or Path(f".logs/{self.run_id}") - self.set_stacks() self.set_envs() self.collect_tfvars() self.collect_gitlab_variables() - def set_stacks(self): - """Set the stacks.""" - self.stacks = STACKS_CHOICES[self.environments_distribution] + def _env(self, name, slug, url, basic_auth_enabled): + host = (url or "").removeprefix("https://").removeprefix("http://").rstrip("/") + return { + "basic_auth_enabled": basic_auth_enabled, + "name": name, + "slug": slug, + "cluster_slug": self.env_to_cluster[name], + "host": host, + "url": url, + } def set_envs(self): """Set the envs.""" self.envs = [ - { - "basic_auth_enabled": True, - "name": DEV_ENV_NAME, - "slug": DEV_ENV_SLUG, - "stack_slug": DEV_ENV_STACK_CHOICES.get( - self.environments_distribution, DEV_STACK_SLUG - ), - "url": self.project_url_dev, - }, - { - "basic_auth_enabled": True, - "name": STAGE_ENV_NAME, - "slug": STAGE_ENV_SLUG, - "stack_slug": STAGE_ENV_STACK_CHOICES.get( - self.environments_distribution, STAGE_STACK_SLUG - ), - "url": self.project_url_stage, - }, - { - "basic_auth_enabled": False, - "name": PROD_ENV_NAME, - "slug": PROD_ENV_SLUG, - "stack_slug": PROD_ENV_STACK_CHOICES.get( - self.environments_distribution, MAIN_STACK_SLUG - ), - "url": self.project_url_prod, - }, + self._env(DEV_ENV_NAME, DEV_ENV_SLUG, self.project_url_dev, True), + self._env(STAGE_ENV_NAME, STAGE_ENV_SLUG, self.project_url_stage, True), + self._env(PROD_ENV_NAME, PROD_ENV_SLUG, self.project_url_prod, False), ] def register_gitlab_variable( @@ -214,7 +198,7 @@ def collect_tfvars(self): self.register_environment_tfvars( ("environment", env["name"]), ("project_url", env["url"]), - ("stack_slug", env["stack_slug"]), + ("cluster_slug", env["cluster_slug"]), env_slug=env["slug"], ) @@ -239,17 +223,24 @@ def init_service(self): cookiecutter( os.path.dirname(os.path.dirname(__file__)), extra_context={ - "deployment_type": self.deployment_type, "internal_service_port": self.internal_service_port, "media_storage": self.media_storage, + "minos_service_image": self.minos_service_image, + "opentofu_component_version": self.opentofu_component_version, + "opentofu_version": self.opentofu_version, "project_dirname": self.project_dirname, "project_name": self.project_name, "project_slug": self.project_slug, - "resources": {"envs": self.envs, "stacks": self.stacks}, + "python_version": self.python_version, + "resources": {"envs": self.envs}, "service_slug": self.service_slug, "terraform_backend": self.terraform_backend, "terraform_cloud_organization": self.terraform_cloud_organization, "tfvars": self.tfvars, + "use_postgres": self.use_postgres and "true" or "false", + "postgres_create_database": ( + self.postgres_create_database and "true" or "false" + ), "use_redis": self.use_redis and "true" or "false", "use_vault": self.vault_url and "true" or "false", }, @@ -281,29 +272,6 @@ def format_files(self): ] ) - def compile_requirements(self): - """Compile the requirements files.""" - click.echo(info("...compiling the requirements files")) - requirements_path = self.service_dir / "requirements" - PIP_COMPILE = [ - "python3", - "-m", - "piptools", - "compile", - "--generate-hashes", - "--no-header", - "--quiet", - "--resolver=backtracking", - "--strip-extras", - "--upgrade", - "--output-file", - ] - for in_file in requirements_path.glob("*.in"): - output_filename = f"{in_file.stem}.txt" - output_file = requirements_path / output_filename - subprocess.run(PIP_COMPILE + [output_file, in_file]) # nosec B603 B607 - click.echo(info(f"\t- {output_filename}")) - def create_static_directory(self): """Create the static directory.""" click.echo(info("...creating the '/static' directory")) @@ -322,7 +290,10 @@ def init_terraform_cloud(self): "TF_VAR_create_organization": self.terraform_cloud_organization_create and "true" or "false", - "TF_VAR_environments": json.dumps(list(map(itemgetter("slug"), self.envs))), + "TF_VAR_create_project": self.terraform_cloud_project_create + and "true" + or "false", + "TF_VAR_environments": json.dumps(list(map(itemgetter("name"), self.envs))), "TF_VAR_hostname": self.terraform_cloud_hostname, "TF_VAR_organization_name": self.terraform_cloud_organization, "TF_VAR_project_name": self.project_name, @@ -527,7 +498,6 @@ def run(self): self.init_service() self.create_env_file() self.format_files() - self.compile_requirements() self.create_static_directory() self.media_storage == "local" and self.create_media_directory() if self.terraform_backend == TERRAFORM_BACKEND_TFC: diff --git a/cookiecutter.json b/cookiecutter.json index 937ebdbe..4f99bf03 100755 --- a/cookiecutter.json +++ b/cookiecutter.json @@ -5,37 +5,36 @@ "project_dirname": "backend", "django_settings_dirname": "{{ cookiecutter.project_slug | slugify(separator='') }}", "internal_service_port": "8000", - "deployment_type": ["digitalocean-k8s", "other-k8s"], "terraform_backend": "gitlab", "terraform_cloud_organization": "", "media_storage": ["digitalocean-s3", "other-s3", "local", "none"], "use_redis": "false", + "use_postgres": "true", + "postgres_create_database": "true", "use_vault": "false", - "environments_distribution": "1", + "python_version": "3.14", + "minos_service_image": "registry.gitlab.com/20tab-open/minos/service:latest", + "opentofu_component_version": "3.11.0", + "opentofu_version": "1.10.6", "resources": { - "stacks": [ - [ - { - "name": "main", - "slug": "main" - } - ] - ], "envs": [ { "name": "development", "slug": "dev", - "stack_slug": "main" + "cluster_slug": "main", + "host": "" }, { "name": "staging", "slug": "stage", - "stack_slug": "main" + "cluster_slug": "main", + "host": "" }, { "name": "production", "slug": "prod", - "stack_slug": "main" + "cluster_slug": "main", + "host": "" } ] }, diff --git a/start.py b/start.py index e9c4d554..0052f9f7 100755 --- a/start.py +++ b/start.py @@ -7,8 +7,6 @@ from bootstrap.collector import Collector from bootstrap.constants import ( - DEPLOYMENT_TYPE_CHOICES, - ENVIRONMENTS_DISTRIBUTION_CHOICES, GITLAB_TOKEN_ENV_VAR, MEDIA_STORAGE_CHOICES, VAULT_TOKEN_ENV_VAR, @@ -33,10 +31,6 @@ @click.option("--project-dirname") @click.option("--service-slug", callback=slugify_option) @click.option("--internal-service-port", default=8000, type=int) -@click.option( - "--deployment-type", - type=click.Choice(DEPLOYMENT_TYPE_CHOICES, case_sensitive=False), -) @click.option("--terraform-backend") @click.option("--terraform-cloud-hostname") @click.option("--terraform-cloud-token") @@ -49,9 +43,6 @@ @click.option("--terraform-cloud-admin-email") @click.option("--vault-token", envvar=VAULT_TOKEN_ENV_VAR) @click.option("--vault-url") -@click.option( - "--environments-distribution", type=click.Choice(ENVIRONMENTS_DISTRIBUTION_CHOICES) -) @click.option("--project-url-dev") @click.option("--project-url-stage") @click.option("--project-url-prod") diff --git a/terraform/terraform-cloud/main.tf b/terraform/terraform-cloud/main.tf index 0ef44527..27079d79 100644 --- a/terraform/terraform-cloud/main.tf +++ b/terraform/terraform-cloud/main.tf @@ -1,39 +1,19 @@ locals { organization = var.create_organization ? tfe_organization.main[0] : data.tfe_organization.main[0] + project = var.create_project ? tfe_project.main[0] : data.tfe_project.main[0] - workspaces = concat( - flatten( - [ - for stage in ["base", "cluster"] : - [ - for stack in var.stacks : - { - name = "${var.project_slug}_${var.service_slug}_${stage}_${stack}" - description = "${var.project_name} project, ${var.service_slug} service, ${stack} stack, ${stage} stage" - tags = [ - "project:${var.project_slug}", - "service:${var.service_slug}", - "stage:${stage}", - "stack:${stack}", - ] - } - ] + workspaces = [ + for env in var.environments : { + name = "${var.project_slug}_${var.service_slug}_${env}" + description = "${var.project_name} ${var.service_slug} service, ${env} environment." + tags = [ + "project:${var.project_slug}", + "layer:service", + "service:${var.service_slug}", + "environment:${env}", ] - ), - [ - for env in var.environments : - { - name = "${var.project_slug}_${var.service_slug}_environment_${env}" - description = "${var.project_name} project, ${var.service_slug} service, ${env} environment" - tags = [ - "project:${var.project_slug}", - "service:${var.service_slug}", - "stage:environment", - "env:${env}", - ] - } - ] - ) + } + ] } terraform { @@ -43,7 +23,7 @@ terraform { required_providers { tfe = { source = "hashicorp/tfe" - version = "~> 0.53" + version = "~> 0.70" } } } @@ -68,20 +48,32 @@ resource "tfe_organization" "main" { email = var.admin_email } -/* Workspaces */ +/* Project */ -resource "tfe_workspace" "main" { - for_each = { for i in local.workspaces : i.name => i } +data "tfe_project" "main" { + count = var.create_project ? 0 : 1 - name = each.value.name - description = each.value.description - organization = local.organization.name - tag_names = each.value.tags + name = var.project_slug + organization = local.organization.name } -resource "tfe_workspace_settings" "main-settings" { - for_each = tfe_workspace.main +resource "tfe_project" "main" { + count = var.create_project ? 1 : 0 + + organization = local.organization.name + name = var.project_slug + description = "${var.project_name} project workspaces." + default_execution_mode = "local" +} + +/* Workspaces */ + +resource "tfe_workspace" "main" { + for_each = { for i in local.workspaces : i.name => i } - workspace_id = each.value.id - execution_mode = "local" + name = each.value.name + description = each.value.description + organization = local.organization.name + project_id = local.project.id + tag_names = each.value.tags } diff --git a/terraform/terraform-cloud/variables.tf b/terraform/terraform-cloud/variables.tf index 79420157..0ca38688 100644 --- a/terraform/terraform-cloud/variables.tf +++ b/terraform/terraform-cloud/variables.tf @@ -10,6 +10,12 @@ variable "create_organization" { default = false } +variable "create_project" { + description = "Tell if the Terraform Cloud project should be created (false when Talos parent has already created it)." + type = bool + default = true +} + variable "environments" { description = "The list of environment slugs." type = list(string) @@ -42,12 +48,6 @@ variable "service_slug" { type = string } -variable "stacks" { - description = "The list of stack slugs." - type = list(string) - default = [] -} - variable "terraform_cloud_token" { description = "The Terraform Cloud token." type = string diff --git a/tests/test_collector.py b/tests/test_collector.py index 1cafa5ff..b8e496dc 100644 --- a/tests/test_collector.py +++ b/tests/test_collector.py @@ -30,103 +30,35 @@ def test_collect(self): collector = Collector( project_name="project_name", ) - collector.set_project_dirname = mock.MagicMock() collector.set_project_slug = mock.MagicMock() collector.set_service_slug = mock.MagicMock() - collector.set_project_urls = mock.MagicMock() collector.set_project_dirname = mock.MagicMock() collector.set_service_dir = mock.MagicMock() collector.set_use_redis = mock.MagicMock() + collector.set_postgres = mock.MagicMock() collector.set_terraform = mock.MagicMock() collector.set_vault = mock.MagicMock() - collector.set_deployment_type = mock.MagicMock() - collector.set_environments_distribution = mock.MagicMock() + collector.set_env_to_cluster = mock.MagicMock() collector.set_project_urls = mock.MagicMock() collector.set_sentry = mock.MagicMock() collector.set_gitlab = mock.MagicMock() collector.set_media_storage = mock.MagicMock() + collector.set_versions = mock.MagicMock() collector.collect() collector.set_project_slug.assert_called_once() + collector.set_service_slug.assert_called_once() collector.set_project_dirname.assert_called_once() collector.set_service_dir.assert_called_once() collector.set_use_redis.assert_called_once() + collector.set_postgres.assert_called_once() collector.set_terraform.assert_called_once() collector.set_vault.assert_called_once() - collector.set_deployment_type.assert_called_once() - collector.set_environments_distribution.assert_called_once() + collector.set_env_to_cluster.assert_called_once() collector.set_project_urls.assert_called_once() collector.set_sentry.assert_called_once() collector.set_gitlab.assert_called_once() collector.set_media_storage.assert_called_once() - - def test_environments_distribution_for_other_k8s_deployment(self): - """Test collecting the environments distribution for other-k8s deployment.""" - collector = Collector(project_name="project_name", deployment_type="other-k8s") - self.assertIsNone(collector.environments_distribution) - with mock.patch("bootstrap.collector.click.prompt") as mocked_prompt: - collector.set_environments_distribution() - self.assertEqual(collector.environments_distribution, "1") - mocked_prompt.assert_not_called() - - def test_environments_distribution_from_default(self): - """Test collecting the environments distribution from its default value.""" - collector = Collector( - project_name="project_name", - ) - self.assertIsNone(collector.environments_distribution) - with mock_input(""): - collector.set_environments_distribution() - self.assertEqual(collector.environments_distribution, "1") - - def test_environments_distribution_from_input(self): - """Test collecting the environments distribution from user input.""" - collector = Collector( - project_name="project_name", - ) - self.assertIsNone(collector.environments_distribution) - with mock_input("one", "yet-another-bad-value", "3"): - collector.set_environments_distribution() - self.assertEqual(collector.environments_distribution, "3") - - def test_environments_distribution_from_options(self): - """Test collecting the environments distribution from the collected options.""" - collector = Collector( - project_name="project_name", environments_distribution="2" - ) - self.assertEqual(collector.environments_distribution, "2") - with mock.patch("bootstrap.collector.click.prompt") as mocked_prompt: - collector.set_environments_distribution() - self.assertEqual(collector.environments_distribution, "2") - mocked_prompt.assert_not_called() - - def test_deployment_type_from_default(self): - """Test collecting the deployment type from its default value.""" - collector = Collector( - project_name="project_name", - ) - self.assertIsNone(collector.deployment_type) - with mock_input(""): - collector.set_deployment_type() - self.assertEqual(collector.deployment_type, "digitalocean-k8s") - - def test_deployment_type_from_input(self): - """Test collecting the deployment type from user input.""" - collector = Collector( - project_name="project_name", - ) - self.assertIsNone(collector.deployment_type) - with mock_input("bad-deployment-type", "yet-another-bad-value", "other-k8s"): - collector.set_deployment_type() - self.assertEqual(collector.deployment_type, "other-k8s") - - def test_deployment_type_from_options(self): - """Test collecting the deployment type from the collected options.""" - collector = Collector(project_name="project_name", deployment_type="other-k8s") - self.assertEqual(collector.deployment_type, "other-k8s") - with mock.patch("bootstrap.collector.click.prompt") as mocked_prompt: - collector.set_deployment_type() - self.assertEqual(collector.deployment_type, "other-k8s") - mocked_prompt.assert_not_called() + collector.set_versions.assert_called_once() def test_gitlab_no(self): """Test not setting Gitlab.""" @@ -304,8 +236,6 @@ def test_launch_runner(self): def test_get_runner(self): """Test getting the runner.""" collector = Collector( - deployment_type="digitalocean-k8s", - environments_distribution="1", internal_service_port=8000, media_storage="local", project_dirname="project_dirname", @@ -317,12 +247,21 @@ def test_get_runner(self): service_slug="django", terraform_backend="terraform-cloud", use_redis=False, + use_postgres=True, + postgres_create_database=True, + env_to_cluster={ + "development": "dev", + "staging": "dev", + "production": "main", + }, + python_version="3.14", + minos_service_image="registry.gitlab.com/20tab-open/minos/service:latest", + opentofu_component_version="3.11.0", + opentofu_version="1.10.6", ) collector._service_dir = Path(".") runner = collector.get_runner() - self.assertEqual(runner.deployment_type, "digitalocean-k8s") - self.assertEqual(runner.environments_distribution, "1") self.assertEqual(runner.internal_service_port, 8000) self.assertEqual(runner.media_storage, "local") self.assertEqual(runner.project_dirname, "project_dirname") diff --git a/{{cookiecutter.project_dirname}}/.gitlab-ci.yml b/{{cookiecutter.project_dirname}}/.gitlab-ci.yml index 552a850b..99e3be8c 100644 --- a/{{cookiecutter.project_dirname}}/.gitlab-ci.yml +++ b/{{cookiecutter.project_dirname}}/.gitlab-ci.yml @@ -1,22 +1,37 @@ +{% set env_dev = cookiecutter.resources.envs|selectattr("slug", "equalto", "dev")|first %}{% set env_stage = cookiecutter.resources.envs|selectattr("slug", "equalto", "stage")|first %}{% set env_prod = cookiecutter.resources.envs|selectattr("slug", "equalto", "prod")|first %}include: + - component: ${CI_SERVER_FQDN}/components/opentofu/apply@{{ cookiecutter.opentofu_component_version }} + inputs: + as: .apply + version: {{ cookiecutter.opentofu_component_version }} + opentofu_version: {{ cookiecutter.opentofu_version }} + no_plan: true + stages: + - Build - Test - Pact-verify - Pact-check - - Build - Deploy - Pact-tag - Report - Sentry variables: + IMAGE_TAG: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} + BUILDAH_FORMAT: docker + BUILDAH_IMAGE: quay.io/buildah/stable:latest + BUILDAH_ISOLATION: chroot + STORAGE_DRIVER: vfs COMPOSE_DOCKER_CLI_BUILD: 1 DOCKER_BUILDKIT: 1 - PACT_PROVIDER_NAME: {{ cookiecutter.project_slug }}-{{ cookiecutter.service_slug }} PROJECT_SLUG: {{ cookiecutter.project_slug }} - SENTRY_PROJECT_NAME: {{ cookiecutter.project_slug }}-{{ cookiecutter.service_slug }} + SERVICE_SLUG: {{ cookiecutter.service_slug }} VERSION_BEFORE_REF: ${CI_COMMIT_BEFORE_SHA} - VERSION_REF: ${CI_COMMIT_SHA} -{% with env=cookiecutter.resources.envs[0] %} + VAULT_ROLE: service-gitlab-job + +# [Environments] +# ----------------------------------------------------------------------------- + .development: rules: &development-rules - &pipeline-push-rule @@ -25,78 +40,63 @@ variables: - &development-rule if: $CI_COMMIT_BRANCH == "develop" variables: - ENV_SLUG: {{ env.slug }} - STACK_SLUG: {{ env.stack_slug }} - VAULT_ROLE: {{ cookiecutter.service_slug }}-{{ env.slug }} + CLUSTER_SLUG: {{ env_dev.cluster_slug }} environment: - name: {{ env.name }}{% if env.url %} - url: {{ env.url }}{% endif %} -{% endwith %}{% with env=cookiecutter.resources.envs[1] %} + name: development + url: {{ env_dev.url }} + .staging: rules: &staging-rules - <<: *pipeline-push-rule - &staging-rule if: $CI_COMMIT_BRANCH == "main" variables: - ENV_SLUG: {{ env.slug }} - STACK_SLUG: {{ env.stack_slug }} - VAULT_ROLE: {{ cookiecutter.service_slug }}-{{ env.slug }} + CLUSTER_SLUG: {{ env_stage.cluster_slug }} environment: - name: {{ env.name }}{% if env.url %} - url: {{ env.url }}{% endif %} -{% endwith %}{% with env=cookiecutter.resources.envs[2] %} + name: staging + url: {{ env_stage.url }} + .production: rules: &production-rules - <<: *pipeline-push-rule - &production-rule if: $CI_COMMIT_TAG variables: - ENV_SLUG: {{ env.slug }} - STACK_SLUG: {{ env.stack_slug }} - VAULT_ROLE: {{ cookiecutter.service_slug }}-{{ env.slug }} + CLUSTER_SLUG: {{ env_prod.cluster_slug }} environment: - name: {{ env.name }}{% if env.url %} - url: {{ env.url }}{% endif %} -{% endwith %} + name: production + url: {{ env_prod.url }} + +# [Pre] +# ----------------------------------------------------------------------------- + .sentry: stage: .pre - image: docker:20 - services: - - docker:20-dind{% if cookiecutter.use_vault == "true" %} + image: + name: getsentry/sentry-cli:latest + entrypoint: [] id_tokens: VAULT_ID_TOKEN: - aud: ${VAULT_ADDR}{% endif %} - script: - - > - docker run --rm - -v ${PWD}:${PWD} - -w ${PWD} - -e CI_ENVIRONMENT_NAME{% if cookiecutter.use_vault == "true" %} - -e ENV_NAME=${CI_ENVIRONMENT_NAME}{% endif %} - -e PROJECT_DIR=${CI_PROJECT_DIR} - -e PROJECT_SLUG - -e RELEASE_END - -e RELEASE_START{% if cookiecutter.use_vault == "false" %} - -e SENTRY_AUTH_TOKEN - -e SENTRY_DSN{% endif %} - -e SENTRY_ORG - -e SENTRY_PROJECT_NAME - -e SENTRY_URL{% if cookiecutter.use_vault == "true" %} - -e SERVICE_SLUG={{ cookiecutter.service_slug }} - -e VAULT_ADDR - -e VAULT_ID_TOKEN - -e VAULT_ROLE{% endif %} - -e VERSION_REF - --entrypoint="" - getsentry/sentry-cli:latest ./scripts/ci_sentry.sh ${SENTRY_CMD} + aud: ${VAULT_ADDR} + variables: + ENV_NAME: ${CI_ENVIRONMENT_SLUG} + PROJECT_DIR: ${CI_PROJECT_DIR} + SENTRY_PROJECT_NAME: ${PROJECT_SLUG}-${SERVICE_SLUG} + before_script: + - source ./scripts/ci_sentry.sh + +# [Sentry Release] +# ----------------------------------------------------------------------------- .sentry_release: extends: - .sentry - variables: - SENTRY_CMD: release - before_script: - - RELEASE_START=$(date +%s) + script: + - > + sentry-cli releases new "${CI_COMMIT_SHA}" + --log-level=debug --project "${SENTRY_PROJECT_NAME}" + - sentry-cli releases set-commits "${CI_COMMIT_SHA}" --auto --ignore-missing + - sentry-cli releases finalize "${CI_COMMIT_SHA}" sentry_release_development: extends: @@ -124,101 +124,108 @@ sentry_release_production: - <<: *sentry-rule - *production-rules +# [Build] +# ----------------------------------------------------------------------------- + +build: + stage: Build + image: ${BUILDAH_IMAGE} + before_script: + - echo "${CI_REGISTRY_PASSWORD}" | buildah login "${CI_REGISTRY}" --username "${CI_REGISTRY_USER}" --password-stdin + - | + TAGS="--tag=${IMAGE_TAG}" + if [ -n "$CI_COMMIT_TAG" ]; then + TAGS="$TAGS --tag=${CI_REGISTRY_IMAGE}:v${CI_COMMIT_TAG} --tag=${CI_REGISTRY_IMAGE}:latest" + fi + script: + - buildah bud --format="${BUILDAH_FORMAT}" --target=remote --layers $TAGS . + - buildah push --all "${IMAGE_TAG}" + after_script: + - buildah logout ${CI_REGISTRY} + +# [Test] +# ----------------------------------------------------------------------------- + +.test: + image: + entrypoint: ["./scripts/ci_test_entrypoint.sh"] + name: ${IMAGE_TAG} + services: + - alias: postgres + name: postgres:17-alpine + variables: + POSTGRES_DB: {{ cookiecutter.project_slug }} + POSTGRES_INITDB_ARGS: --no-sync + POSTGRES_PASSWORD: postgres + test: + extends: + - .test stage: Test - image: docker:20 - services: - - docker:20-dind - needs: [] rules: - if: $CI_PIPELINE_SOURCE == "push" - variables: - {{ cookiecutter.service_slug|upper }}_CONTAINER_NAME: "${CI_PROJECT_PATH_SLUG}-${CI_JOB_NAME}-${CI_JOB_ID}_{{ cookiecutter.service_slug }}" - {{ cookiecutter.service_slug|upper }}_BUILD_TARGET: "test" - {{ cookiecutter.service_slug|upper }}_IMAGE_NAME: "gitlabci_{{ cookiecutter.project_slug }}_{{ cookiecutter.service_slug }}" - {{ cookiecutter.service_slug|upper }}_IMAGE_TAG: "${CI_JOB_NAME}-${CI_JOB_ID}" - COMPOSE_PROJECT_NAME: "${CI_PROJECT_PATH_SLUG}-${CI_JOB_NAME}-${CI_JOB_ID}" script: - - docker-compose build - - docker-compose run --name ${{ "{" }}{{ cookiecutter.service_slug|upper }}_CONTAINER_NAME} {{ cookiecutter.service_slug }} - - docker cp ${{ "{" }}{{ cookiecutter.service_slug|upper }}_CONTAINER_NAME}:/app/htmlcov htmlcov - after_script: - - docker-compose down -v + - ./scripts/test.sh coverage: '/^TOTAL.*\s+(\d+\%)$/' artifacts: expire_in: 1 day paths: - htmlcov + reports: + coverage_report: + coverage_format: cobertura + path: .artifacts/cobertura.xml + junit: .artifacts/junit.xml when: always -pact-verify-test: +# [Pact Verify] +# ----------------------------------------------------------------------------- + +.pact-verify: + extends: + - .test stage: Pact-verify - image: docker:20 - services: - - docker:20-dind{% if cookiecutter.use_vault == "true" %} id_tokens: VAULT_ID_TOKEN: - aud: ${VAULT_ADDR}{% endif %} - needs: [] - variables: - {{ cookiecutter.service_slug|upper }}_BUILD_TARGET: "test" - {{ cookiecutter.service_slug|upper }}_IMAGE_NAME: "gitlabci_{{ cookiecutter.project_slug }}_{{ cookiecutter.service_slug }}" - {{ cookiecutter.service_slug|upper }}_IMAGE_TAG: "${CI_JOB_NAME}-${CI_JOB_ID}" - COMPOSE_PROJECT_NAME: "${CI_PROJECT_PATH_SLUG}-${CI_JOB_NAME}-${CI_JOB_ID}" + aud: ${VAULT_ADDR} + +pact-verify-test: + extends: + - .pact-verify rules: - <<: *pipeline-push-rule - if: $PACT_ENABLED == "true" allow_failure: true - before_script: &pact-verify-before-script{% if cookiecutter.use_vault == "true" %} - - > - vault_token=$(wget --quiet --post-data="role=pact&jwt=${VAULT_ID_TOKEN}" - "${VAULT_ADDR%/}"/v1/auth/gitlab-jwt/login -O - | - sed -n 's/^.*"client_token":"\([^"]*\)".*$/\1/p') - - > - PACT_BROKER_AUTH_URL=$(wget --quiet --header="X-Vault-Token: ${vault_token}" - "${VAULT_ADDR%/}"/v1/"${PROJECT_SLUG}"/pact -O - | - sed -n 's/^.*"pact_broker_auth_url":"\([^"]*\)".*$/\1/p'){% endif %} - - export PACT_BROKER_URL="${PACT_BROKER_AUTH_URL}" script: - > if [ "${CI_COMMIT_BRANCH}" ]; then PACT_CONSUMER_TAG="branch:${CI_COMMIT_BRANCH}"; - else PACT_CONSUMER_TAG="tag:${CI_COMMIT_TAG}"; + else PACT_CONSUMER_TAG="${CI_COMMIT_TAG}"; fi - - docker-compose build - > - docker-compose run --rm {{ cookiecutter.service_slug }} ./scripts/pact_verify.sh - --pact-provider-version=${VERSION_REF} - --pact-verify-consumer-tag=${PACT_CONSUMER_TAG} - --pact-verify-consumer-tag="env:dev" - --pact-verify-consumer-tag="env:stage" - --pact-verify-consumer-tag="env:prod" + ./scripts/pact_verify.sh + --pact-provider-version="${CI_COMMIT_SHA}" + --pact-verify-consumer-tag="${PACT_CONSUMER_TAG}" + --pact-verify-consumer-tag="development" + --pact-verify-consumer-tag="staging" + --pact-verify-consumer-tag="production" --pact-publish-results - after_script: - - docker-compose down -v pact-verify-webhook: - stage: Pact-verify - image: docker:20 - services: - - docker:20-dind - variables: - COMPOSE_PROJECT_NAME: "${CI_PROJECT_PATH_SLUG}-${CI_JOB_NAME}-${CI_JOB_ID}" - {{ cookiecutter.service_slug|upper }}_IMAGE_NAME: "gitlabci_{{ cookiecutter.project_slug }}_{{ cookiecutter.service_slug }}" - {{ cookiecutter.service_slug|upper }}_BUILD_TARGET: "test" + extends: + - .pact-verify rules: - if: $CI_PIPELINE_SOURCE == "trigger" && $PACT_ENABLED == "true" allow_failure: true - before_script: *pact-verify-before-script script: - - docker-compose build - > - docker-compose run --rm {{ cookiecutter.service_slug }} ./scripts/pact_verify.sh - --pact-provider-version=${VERSION_REF} - --pact-verify-consumer-tag=${PACT_CONSUMER_TAG} + ./scripts/pact_verify.sh + --pact-provider-version="${CI_COMMIT_SHA}" + --pact-verify-consumer-tag="${PACT_CONSUMER_TAG}" --pact-publish-results - after_script: - - docker-compose down -v + +# [Report] +# ----------------------------------------------------------------------------- pages: stage: Report @@ -234,40 +241,30 @@ pages: paths: - public +# [Pact - Can I Deploy] +# ----------------------------------------------------------------------------- + .pact: image: - name: docker:20 - services: - - docker:20-dind{% if cookiecutter.use_vault == "true" %} + name: pactfoundation/pact-cli:latest + entrypoint: [] id_tokens: VAULT_ID_TOKEN: - aud: ${VAULT_ADDR}{% endif %} - script: - - > - docker run --rm - -v ${PWD}:${PWD} - -w ${PWD}{% if cookiecutter.use_vault == "true" %} - -e ENV_SLUG{% else %} - -e PACT_BROKER_BASE_URL - -e PACT_BROKER_PASSWORD - -e PACT_BROKER_USERNAME{% endif %} - -e PROJECT_SLUG{% if cookiecutter.use_vault == "true" %} - -e VAULT_ADDR - -e VAULT_ID_TOKEN{% endif %} - --entrypoint="" - pactfoundation/pact-cli:latest-node14 ./scripts/ci_pact.sh ${PACT_CMD} + aud: ${VAULT_ADDR} + before_script: + - source ./scripts/ci_pact.sh .can-i-deploy: extends: - .pact stage: Pact-check needs: ["pact-verify-test"] - before_script: + script: - > - export PACT_CMD="can-i-deploy - --pacticipant ${PACT_PROVIDER_NAME} - --version ${VERSION_REF} - --to env:${ENV_SLUG}" + pact-broker can-i-deploy + --pacticipant "${PACT_PROVIDER_NAME}" + --version "${CI_COMMIT_SHA}" + --to "${CI_ENVIRONMENT_SLUG}" can-i-deploy_development: extends: @@ -295,134 +292,67 @@ can-i-deploy_production: - <<: *skip-pact-rule - *production-rules -.build: - stage: Build - image: docker:20 - services: - - docker:20-dind +# [Deploy] +# ----------------------------------------------------------------------------- + +.deploy: + stage: Deploy + extends: + - .apply + id_tokens: + VAULT_ID_TOKEN: + aud: ${VAULT_ADDR} + image: {{ cookiecutter.minos_service_image }} + variables: + GITLAB_TOFU_INIT_NO_RECONFIGURE: true + GITLAB_TOFU_ROOT_DIR: ${CI_PROJECT_DIR}/tofu + PROJECT_DIR: ${CI_PROJECT_DIR} + TF_CLOUD_HOSTNAME: app.terraform.io + TF_CLOUD_ORGANIZATION: {{ cookiecutter.terraform_cloud_organization }} + TF_VAR_image: ${IMAGE_TAG} + TF_VAR_registry_password: ${CI_DEPLOY_PASSWORD} + TF_VAR_registry_server: ${CI_REGISTRY} + TF_VAR_registry_username: ${CI_DEPLOY_USER} + TF_WORKSPACE: "${PROJECT_SLUG}_${SERVICE_SLUG}_${CI_ENVIRONMENT_SLUG}" + TOFU_BACKEND: terraform-cloud + TOFU_VAR_FILES: "common.tfvars ${CI_ENVIRONMENT_SLUG}/this.tfvars" + VAULT_SECRETS_PREFIX: "envs/${CI_ENVIRONMENT_SLUG}" + VAULT_SECRETS: "digitalocean" + VAULT_SERVICE_SECRETS: "shared-secrets.tftpl" before_script: - - export DOCKER_CONFIG=${PWD}/.dockerconfig - - docker login --username "${CI_REGISTRY_USER}" --password "${CI_REGISTRY_PASSWORD}" "${CI_REGISTRY}" - script: - - docker build -t ${CI_REGISTRY}/${CI_PROJECT_PATH}:${VERSION_REF} --target remote --pull . - - docker push ${CI_REGISTRY}/${CI_PROJECT_PATH}:${VERSION_REF} - after_script: - - docker logout ${CI_REGISTRY} + - export TF_CLI_ARGS="${TOFU_VAR_FILE_ARGS}" -build_development: +deploy_development: extends: + - .deploy - .development - - .build needs: - job: can-i-deploy_development optional: true - job: test -build_staging: +deploy_staging: extends: + - .deploy - .staging - - .build needs: - job: can-i-deploy_staging optional: true - job: test -build_production: +deploy_production: extends: + - .deploy - .production - - .build needs: - job: can-i-deploy_production optional: true - job: test -.deploy: - stage: Deploy - image: - name: docker:20 - services: - - docker:20-dind{% if cookiecutter.use_vault == "true" %} - id_tokens: - VAULT_ID_TOKEN: - aud: ${VAULT_ADDR}{% endif %} - variables: - TF_ROOT: ${CI_PROJECT_DIR}/terraform/{{ cookiecutter.deployment_type }} - before_script: - - export TF_VAR_service_container_image=${CI_REGISTRY_IMAGE}:${VERSION_REF} - script: - - > - docker run --rm - -u `id -u` - -v ${PWD}:${PWD} - -w ${PWD}{% if cookiecutter.terraform_backend == "gitlab" %} - -e CI_API_V4_URL - -e CI_COMMIT_SHA - -e CI_JOB_ID - -e CI_JOB_STAGE - -e CI_JOB_TOKEN - -e CI_PROJECT_ID - -e CI_PROJECT_NAME - -e CI_PROJECT_NAMESPACE - -e CI_PROJECT_PATH - -e CI_PROJECT_URL{% endif %} - -e ENV_SLUG - -e PROJECT_DIR=${CI_PROJECT_DIR} - -e PROJECT_SLUG - -e STACK_SLUG - -e TERRAFORM_BACKEND={{ cookiecutter.terraform_backend }} - -e TERRAFORM_EXTRA_VAR_FILE=${ENV_SLUG}.tfvars - -e TERRAFORM_VARS_DIR=${CI_PROJECT_DIR}/terraform/vars - -e TF_ROOT{% if cookiecutter.terraform_backend == "gitlab" %} - -e TF_STATE_NAME="env_${ENV_SLUG}"{% endif %}{% if cookiecutter.use_vault == "false" %}{% if cookiecutter.deployment_type == "digitalocean-k8s" %} - -e TF_VAR_digitalocean_token="${DIGITALOCEAN_TOKEN}"{% endif %} - -e TF_VAR_email_url="${EMAIL_URL}" - -e TF_VAR_service_slug="{{ cookiecutter.service_slug }}"{% if cookiecutter.deployment_type == "other-k8s" %} - -e TF_VAR_kubernetes_cluster_ca_certificate="${KUBERNETES_CLUSTER_CA_CERTIFICATE}" - -e TF_VAR_kubernetes_host="${KUBERNETES_HOST}" - -e TF_VAR_kubernetes_token="${KUBERNETES_TOKEN}"{% endif %}{% if "s3" in cookiecutter.media_storage %} - -e TF_VAR_s3_access_id="${S3_ACCESS_ID}" - -e TF_VAR_s3_secret_key="${S3_SECRET_KEY}" - -e TF_VAR_s3_region="${S3_REGION}" - -e TF_VAR_s3_host="${S3_HOST}" - -e TF_VAR_s3_bucket_name="${S3_BUCKET_NAME}"{% endif %} - -e TF_VAR_sentry_dsn="${SENTRY_DSN}"{% endif %} - -e TF_VAR_service_container_image{% if cookiecutter.terraform_backend != "gitlab" %} - -e TF_WORKSPACE="{{ cookiecutter.project_slug }}_backend_environment_${ENV_SLUG}"{% endif %}{% if cookiecutter.terraform_backend == "terraform-cloud" and cookiecutter.use_vault == "false" %} - -e TFC_TOKEN{% endif %}{% if cookiecutter.use_vault == "true" %} - -e VAULT_ADDR - -e VAULT_ID_TOKEN - -e VAULT_ROLE - -e VAULT_SECRETS="digitalocean email k8s s3 {{ cookiecutter.service_slug }}/extra {{ cookiecutter.service_slug }}/sentry" - -e VAULT_SECRETS_PREFIX="envs/${CI_ENVIRONMENT_NAME}" - -e VAULT_VERSION{% endif %} - registry.gitlab.com/gitlab-org/terraform-images/stable:latest ./scripts/deploy.sh - artifacts: - name: plan - reports: - terraform: ${TF_ROOT}/plan.json - -deploy_development: - extends: - - .development - - .deploy - needs: ["build_development"] - -deploy_staging: - extends: - - .staging - - .deploy - needs: ["build_staging"] - -deploy_production: - extends: - - .production - - .deploy - needs: ["build_production"] - .rollback: extends: .deploy - before_script: - - export TF_VAR_service_container_image=${CI_REGISTRY_IMAGE}:${VERSION_BEFORE_REF} + variables: + TF_VAR_image: "${CI_REGISTRY_IMAGE}:${CI_COMMIT_BEFORE_SHA}" rollback_development: extends: @@ -457,19 +387,23 @@ rollback_production: when: manual allow_failure: true +# [Pact - Tag] +# ----------------------------------------------------------------------------- + .create-version-tag: extends: - .pact stage: Pact-tag - before_script: + script: - > - export PACT_CMD="create-version-tag - --pacticipant ${PACT_PROVIDER_NAME} - --version ${VERSION_REF} - --tag env:${ENV_SLUG}" + pact-broker create-version-tag + --pacticipant "${PACT_PROVIDER_NAME}" + --version "${CI_COMMIT_SHA}" + --tag "${CI_ENVIRONMENT_SLUG}" create-version-tag_development: extends: + - .development - .create-version-tag needs: ["deploy_development"] rules: @@ -478,6 +412,7 @@ create-version-tag_development: create-version-tag_staging: extends: + - .staging - .create-version-tag needs: ["deploy_staging"] rules: @@ -486,20 +421,26 @@ create-version-tag_staging: create-version-tag_production: extends: + - .production - .create-version-tag needs: ["deploy_production"] rules: - <<: *skip-pact-rule - *production-rules +# [Sentry] +# ----------------------------------------------------------------------------- + .sentry_deploy_success: extends: - .sentry - variables: - SENTRY_CMD: success stage: Sentry - before_script: + script: + - RELEASE_START=$(date -d "${CI_PIPELINE_CREATED_AT}" +%s) - RELEASE_END=$(date +%s) + - > + sentry-cli releases deploys "${CI_COMMIT_SHA}" + new --env "${ENV_NAME}" --time $((RELEASE_END-RELEASE_START)) sentry_success_development: extends: @@ -537,9 +478,9 @@ sentry_success_production: .sentry_deploy_failure: extends: - .sentry - variables: - SENTRY_CMD: failure stage: Sentry + script: + - sentry-cli send-event --message "Deploy to ${ENV_NAME} failed." sentry_failure_development: extends: diff --git a/{{cookiecutter.project_dirname}}/Dockerfile b/{{cookiecutter.project_dirname}}/Dockerfile index 0167bec8..0f773c08 100644 --- a/{{cookiecutter.project_dirname}}/Dockerfile +++ b/{{cookiecutter.project_dirname}}/Dockerfile @@ -1,78 +1,95 @@ -FROM python:3.12-slim-bookworm AS base +FROM ghcr.io/astral-sh/uv:python{{ cookiecutter.python_version }}-bookworm-slim AS base + +LABEL company="20tab" project="{{ cookiecutter.project_slug }}" service="{{ cookiecutter.service_slug }}" stage="base" + +ARG DEBIAN_FRONTEND=noninteractive \ + GROUP_ID=1000 \ + USER_ID=1000 \ + USER=appuser + +ENV APPUSER=$USER \ + INTERNAL_SERVICE_PORT={{ cookiecutter.internal_service_port }} \ + LANG=C.UTF-8 \ + LC_ALL=C.UTF-8 \ + PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 \ + WORKDIR=/app \ + UV_COMPILE_BYTECODE=1 + +ENV UV_PROJECT_ENVIRONMENT="/home/$APPUSER/.venv" \ + UV_PYTHON_INSTALL_DIR="/home/$APPUSER/.local/share/uv/python" + +ENV PATH="$UV_PROJECT_ENVIRONMENT/bin:$PATH" -LABEL company="20tab" project="{{ cookiecutter.project_slug }}" service="backend" stage="base" -ARG DEBIAN_FRONTEND=noninteractive -ARG USER=appuser -ENV APPUSER=$USER LANG=C.UTF-8 LC_ALL=C.UTF-8 PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 WORKDIR=/app WORKDIR $WORKDIR -RUN useradd --skel /dev/null --create-home $APPUSER -RUN chown $APPUSER:$APPUSER $WORKDIR -ENV PATH="/home/${APPUSER}/.local/bin:${PATH}" -ARG PACKAGES_PATH=/home/${APPUSER}/.local/lib/python3.12/site-packages + +RUN groupadd --gid "$GROUP_ID" "$APPUSER" && \ + useradd --uid "$USER_ID" --gid "$GROUP_ID" --create-home --skel /dev/null \ + "$APPUSER" + RUN apt-get update \ && apt-get install --assume-yes --no-install-recommends \ + ca-certificates \ libpq5 \ && rm -rf /var/lib/apt/lists/* -COPY --chown=$APPUSER ./requirements/base.txt requirements/base.txt + +COPY --chown=$APPUSER ./pyproject.toml ./uv.lock ./ + RUN apt-get update \ && apt-get install --assume-yes --no-install-recommends \ gcc \ libc6-dev \ libpq-dev \ - && su $APPUSER -c "python3 -m pip install --user --no-cache-dir -r requirements/base.txt" \ - && find ${PACKAGES_PATH} -regex '^.*/locale/.*/*.\(mo\|po\)$' -not -path '*/en*' -not -path '*/it*' -delete || true \ + && chown -R "$USER_ID":"$GROUP_ID" "$WORKDIR" \ + && su "$APPUSER" -c "uv sync --frozen" \ && apt-get purge --assume-yes --auto-remove \ gcc \ libc6-dev \ libpq-dev \ && rm -rf /var/lib/apt/lists/* -COPY --chown=$APPUSER ./requirements/common.txt requirements/common.txt -RUN su $APPUSER -c "python3 -m pip install --user --no-cache-dir -r requirements/common.txt" \ - && find ${PACKAGES_PATH} -regex '^.*/locale/.*/*.\(mo\|po\)$' -not -path '*/en*' -not -path '*/it*' -delete || true -FROM base AS test - -LABEL company="20tab" project="{{ cookiecutter.project_slug }}" service="backend" stage="test" -ENV DJANGO_CONFIGURATION=Testing -USER $APPUSER -COPY --chown=$APPUSER ./requirements/test.txt requirements/test.txt -RUN python3 -m pip install --user --no-cache-dir -r requirements/test.txt -COPY --chown=$APPUSER . . -CMD ./scripts/test.sh FROM base AS remote -LABEL company="20tab" project="{{ cookiecutter.project_slug }}" service="backend" stage="remote" -ENV DJANGO_CONFIGURATION=Remote INTERNAL_SERVICE_PORT={{ cookiecutter.internal_service_port }} -USER $APPUSER -ARG PACKAGES_PATH=/home/${APPUSER}/.local/lib/python3.12/site-packages -COPY --chown=$APPUSER ./requirements/remote.txt requirements/remote.txt -RUN python3 -m pip install --user --no-cache-dir -r requirements/remote.txt \ - && find ${PACKAGES_PATH}/boto*/data/* -maxdepth 0 -type d -not -name s3* -exec rm -rf {} \; || true +LABEL company="20tab" project="{{ cookiecutter.project_slug }}" service="{{ cookiecutter.service_slug }}" stage="remote" + +ENV DJANGO_CONFIGURATION=Remote + +RUN su "$APPUSER" -c "uv sync --frozen --group remote" + COPY --chown=$APPUSER . . + RUN DJANGO_SECRET_KEY=build python3 -m manage collectstatic --clear --link --noinput + +USER $APPUSER + ENTRYPOINT ["./scripts/entrypoint.sh"] -CMD ["python3", "-m", "gunicorn", "{{ cookiecutter.django_settings_dirname }}.asgi"] + +CMD ["gunicorn", "{{ cookiecutter.django_settings_dirname }}.asgi"] + FROM base AS local -LABEL company="20tab" project="{{ cookiecutter.project_slug }}" service="backend" stage="local" -ENV DJANGO_CONFIGURATION=Local INTERNAL_SERVICE_PORT={{ cookiecutter.internal_service_port }} +LABEL company="20tab" project="{{ cookiecutter.project_slug }}" service="{{ cookiecutter.service_slug }}" stage="local" + +ENV DJANGO_CONFIGURATION=Local + RUN apt-get update \ && apt-get install --assume-yes --no-install-recommends \ curl \ - gcc \ gettext \ git \ graphviz \ - libpq-dev \ - make \ openssh-client \ - postgresql-client -USER $APPUSER -COPY --chown=$APPUSER ./requirements/local.txt requirements/local.txt -RUN python3 -m pip install --user --no-cache-dir -r requirements/local.txt + postgresql-client \ + && rm -rf /var/lib/apt/lists/* + +RUN su "$APPUSER" -c "uv sync --frozen --group local" + COPY --chown=$APPUSER . . -RUN DJANGO_SECRET_KEY=build python3 -m manage collectstatic --clear --link --noinput + +USER $APPUSER + ENTRYPOINT ["./scripts/entrypoint.sh"] + CMD ["sh", "-c", "exec python3 -m manage runserver 0.0.0.0:${INTERNAL_SERVICE_PORT}"] diff --git a/{{cookiecutter.project_dirname}}/minos/common.tfvars b/{{cookiecutter.project_dirname}}/minos/common.tfvars new file mode 100644 index 00000000..6bc8db30 --- /dev/null +++ b/{{cookiecutter.project_dirname}}/minos/common.tfvars @@ -0,0 +1,9 @@ +deployments = { + {{ cookiecutter.service_slug }} = { + port = "{{ cookiecutter.internal_service_port }}" + } +} +postgres_enabled = {{ cookiecutter.use_postgres }} +postgres_create_database = {{ cookiecutter.postgres_create_database }} +service_slug = "{{ cookiecutter.service_slug }}" +shared_secret_values_json = "shared-secrets.tftpl.json" diff --git a/{{cookiecutter.project_dirname}}/minos/development/shared-config.yaml b/{{cookiecutter.project_dirname}}/minos/development/shared-config.yaml new file mode 100644 index 00000000..8eeeb3e4 --- /dev/null +++ b/{{cookiecutter.project_dirname}}/minos/development/shared-config.yaml @@ -0,0 +1 @@ +INTERNAL_SERVICE_PORT: "{{ cookiecutter.internal_service_port }}" diff --git a/{{cookiecutter.project_dirname}}/minos/development/this.tfvars b/{{cookiecutter.project_dirname}}/minos/development/this.tfvars new file mode 100644 index 00000000..81fef414 --- /dev/null +++ b/{{cookiecutter.project_dirname}}/minos/development/this.tfvars @@ -0,0 +1,14 @@ +{% set env = cookiecutter.resources.envs|selectattr("slug", "equalto", "dev")|first %}certificates = { + primary = { + letsencrypt_email = "tech@20tab.com" + hosts = ["{{ env.host }}"] + } +} +cluster_slug = "{{ cookiecutter.project_slug }}-{{ env.cluster_slug }}" +environment = "{{ env.name }}" +namespace = "{{ cookiecutter.project_slug }}-{{ env.slug }}" +project_slug = "{{ cookiecutter.project_slug }}" +routing = { + "{{ env.host }}" = { deployment = "{{ cookiecutter.service_slug }}" } +} +shared_config_values_yaml = "{{ env.name }}/shared-config.yaml" diff --git a/{{cookiecutter.project_dirname}}/minos/production/shared-config.yaml b/{{cookiecutter.project_dirname}}/minos/production/shared-config.yaml new file mode 100644 index 00000000..8eeeb3e4 --- /dev/null +++ b/{{cookiecutter.project_dirname}}/minos/production/shared-config.yaml @@ -0,0 +1 @@ +INTERNAL_SERVICE_PORT: "{{ cookiecutter.internal_service_port }}" diff --git a/{{cookiecutter.project_dirname}}/minos/production/this.tfvars b/{{cookiecutter.project_dirname}}/minos/production/this.tfvars new file mode 100644 index 00000000..5603c9d2 --- /dev/null +++ b/{{cookiecutter.project_dirname}}/minos/production/this.tfvars @@ -0,0 +1,14 @@ +{% set env = cookiecutter.resources.envs|selectattr("slug", "equalto", "prod")|first %}certificates = { + primary = { + letsencrypt_email = "tech@20tab.com" + hosts = ["{{ env.host }}"] + } +} +cluster_slug = "{{ cookiecutter.project_slug }}-{{ env.cluster_slug }}" +environment = "{{ env.name }}" +namespace = "{{ cookiecutter.project_slug }}-{{ env.slug }}" +project_slug = "{{ cookiecutter.project_slug }}" +routing = { + "{{ env.host }}" = { deployment = "{{ cookiecutter.service_slug }}" } +} +shared_config_values_yaml = "{{ env.name }}/shared-config.yaml" diff --git a/{{cookiecutter.project_dirname}}/minos/staging/shared-config.yaml b/{{cookiecutter.project_dirname}}/minos/staging/shared-config.yaml new file mode 100644 index 00000000..8eeeb3e4 --- /dev/null +++ b/{{cookiecutter.project_dirname}}/minos/staging/shared-config.yaml @@ -0,0 +1 @@ +INTERNAL_SERVICE_PORT: "{{ cookiecutter.internal_service_port }}" diff --git a/{{cookiecutter.project_dirname}}/minos/staging/this.tfvars b/{{cookiecutter.project_dirname}}/minos/staging/this.tfvars new file mode 100644 index 00000000..d29f93cf --- /dev/null +++ b/{{cookiecutter.project_dirname}}/minos/staging/this.tfvars @@ -0,0 +1,14 @@ +{% set env = cookiecutter.resources.envs|selectattr("slug", "equalto", "stage")|first %}certificates = { + primary = { + letsencrypt_email = "tech@20tab.com" + hosts = ["{{ env.host }}"] + } +} +cluster_slug = "{{ cookiecutter.project_slug }}-{{ env.cluster_slug }}" +environment = "{{ env.name }}" +namespace = "{{ cookiecutter.project_slug }}-{{ env.slug }}" +project_slug = "{{ cookiecutter.project_slug }}" +routing = { + "{{ env.host }}" = { deployment = "{{ cookiecutter.service_slug }}" } +} +shared_config_values_yaml = "{{ env.name }}/shared-config.yaml" diff --git a/{{cookiecutter.project_dirname}}/pyproject.toml b/{{cookiecutter.project_dirname}}/pyproject.toml index 38bd4e21..d99c2c3f 100644 --- a/{{cookiecutter.project_dirname}}/pyproject.toml +++ b/{{cookiecutter.project_dirname}}/pyproject.toml @@ -1,5 +1,57 @@ +[project] +name = "{{ cookiecutter.project_slug }}" +version = "0.0.1" +description = "{{ cookiecutter.project_name }}" +readme = "README.md" +requires-python = ">={{ cookiecutter.python_version }}" +classifiers = [ + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: {{ cookiecutter.python_version }}", +] +dependencies = [ + "django[argon2]~=5.2.0", + "django-configurations[cache,database,email]~=2.5.0", + "psycopg~=3.2.0", +] + +[dependency-groups] +test = [ + "bandit[toml]~=1.9.0", + "behave-django~=1.4.0", + "coverage[toml]~=7.13.0", + "model-bakery~=1.23.0", + "mypy~=1.19.0", + "pactman~=2.30.0", + "pytest-django~=4.11.0", + "pytest-dotenv~=0.5.0", + "ruff~=0.14.0", + "tblib~=3.0.0", + "time-machine~=3.2.0", +] +local = [ + {include-group = "test"}, + "django-debug-toolbar~=6.2.0", + "django-extensions~=4.1.0", + "graphviz~=0.20.0", + "ipython~=9.9.0", + "prek~=0.2", + "rust-just~=1.46.0", +] +remote = [ + {%- if "s3" in cookiecutter.media_storage %} + "django-storages[boto3]~=1.14.0", + {%- endif %} + "gunicorn~=23.0.0", + {%- if cookiecutter.use_redis == "true" %} + "redis~=5.0.0", + {%- endif %} + "sentry-sdk~=2.50.0", + "uvicorn[standard]~=0.40.0", + "whitenoise[brotli]~=6.11.0", +] + [tool.ruff] -target-version = "py312" +target-version = "py{{ cookiecutter.python_version | replace('.', '') }}" [tool.ruff.lint] ignore = [ @@ -76,7 +128,7 @@ enable_error_code = [ "truthy-bool", ] ignore_missing_imports = true -python_version = "3.12" +python_version = "{{ cookiecutter.python_version }}" [tool.bandit] exclude_dirs = [ diff --git a/{{cookiecutter.project_dirname}}/requirements/base.in b/{{cookiecutter.project_dirname}}/requirements/base.in deleted file mode 100644 index 2a5c23a4..00000000 --- a/{{cookiecutter.project_dirname}}/requirements/base.in +++ /dev/null @@ -1 +0,0 @@ -psycopg[c]~=3.1.0 diff --git a/{{cookiecutter.project_dirname}}/requirements/common.in b/{{cookiecutter.project_dirname}}/requirements/common.in deleted file mode 100644 index 81474693..00000000 --- a/{{cookiecutter.project_dirname}}/requirements/common.in +++ /dev/null @@ -1,3 +0,0 @@ --r base.in -django-configurations[cache,database,email]~=2.5.0 -django~=5.0.0 diff --git a/{{cookiecutter.project_dirname}}/requirements/local.in b/{{cookiecutter.project_dirname}}/requirements/local.in deleted file mode 100644 index 2cb0bd97..00000000 --- a/{{cookiecutter.project_dirname}}/requirements/local.in +++ /dev/null @@ -1,10 +0,0 @@ --r test.in -django-debug-toolbar~=4.2.0 -django-extensions~=3.2.0 -graphviz~=0.20.0 -ipython~=8.20.0 -pip-tools~=7.3.0 -pre-commit~=3.6.0 -pydot~=2.0.0 -python-dotenv~=1.0.0 -rope~=1.11.0 diff --git a/{{cookiecutter.project_dirname}}/requirements/remote.in b/{{cookiecutter.project_dirname}}/requirements/remote.in deleted file mode 100644 index 502df9fc..00000000 --- a/{{cookiecutter.project_dirname}}/requirements/remote.in +++ /dev/null @@ -1,8 +0,0 @@ --r common.in -argon2-cffi~=23.1.0 -{% if "s3" in cookiecutter.media_storage %}django-storages[boto3]~=1.14.0 -{% endif %}gunicorn~=22.0.0 -{% if cookiecutter.use_redis == "true" %}redis~=5.0.0 -{% endif %}sentry-sdk~=1.39.0 -uvicorn[standard]~=0.25.0 -whitenoise[brotli]~=6.6.0 diff --git a/{{cookiecutter.project_dirname}}/requirements/test.in b/{{cookiecutter.project_dirname}}/requirements/test.in deleted file mode 100644 index 2903b8cb..00000000 --- a/{{cookiecutter.project_dirname}}/requirements/test.in +++ /dev/null @@ -1,12 +0,0 @@ --r common.in -bandit[toml]~=1.7.0 -behave-django~=1.4.0 -coverage[toml]~=7.4.0 -mypy~=1.8.0 -pactman~=2.30.0 -pip-audit~=2.6.0 -pytest-django~=4.7.0 -pytest-dotenv~=0.5.0 -ruff~=0.1.0 -tblib~=3.0.0 -time-machine~=2.13.0 diff --git a/{{cookiecutter.project_dirname}}/scripts/behave.sh b/{{cookiecutter.project_dirname}}/scripts/behave.sh deleted file mode 100755 index 790f0a9a..00000000 --- a/{{cookiecutter.project_dirname}}/scripts/behave.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -python3 -m manage behave --configuration=Testing --format=progress --noinput --simple diff --git a/{{cookiecutter.project_dirname}}/scripts/check.sh b/{{cookiecutter.project_dirname}}/scripts/check.sh index 2119cf3c..2194f0d6 100755 --- a/{{cookiecutter.project_dirname}}/scripts/check.sh +++ b/{{cookiecutter.project_dirname}}/scripts/check.sh @@ -2,10 +2,7 @@ set -euo pipefail -python3 -m manage check -python3 -m manage makemigrations --dry-run --check -python3 -m ruff format --check . -python3 -m ruff check . -python3 -m mypy --no-site-packages . -python3 -m bandit --configfile pyproject.toml --quiet --recursive . -python3 -m pip_audit --require-hashes --disable-pip --requirement requirements/remote.txt +uv run ruff format --check . +uv run ruff check . +uv run mypy . +uv run bandit -c pyproject.toml --quiet --recursive . diff --git a/{{cookiecutter.project_dirname}}/scripts/ci_pact.sh b/{{cookiecutter.project_dirname}}/scripts/ci_pact.sh index 958c5e08..96521b4b 100755 --- a/{{cookiecutter.project_dirname}}/scripts/ci_pact.sh +++ b/{{cookiecutter.project_dirname}}/scripts/ci_pact.sh @@ -1,13 +1,20 @@ #!/usr/bin/env sh -set -e +set -euo pipefail if [ "${VAULT_ADDR}" != "" ]; then apk update && apk add curl jq - vault_token=$(curl --silent --request POST --data "role=pact" --data "jwt=${VAULT_ID_TOKEN}" "${VAULT_ADDR%/}"/v1/auth/gitlab-jwt/login | jq -r .auth.client_token) + vault_token=$(curl --silent --request POST \ + --data "role=pact" \ + --data "jwt=${VAULT_ID_TOKEN}" \ + "${VAULT_ADDR%/}"/v1/auth/gitlab-jwt/login | \ + jq -r .auth.client_token) - pact_secrets=$(curl --silent --header "X-Vault-Token: ${vault_token}" "${VAULT_ADDR%/}"/v1/"${PROJECT_SLUG}"/pact | jq -r .data) + pact_secrets=$(curl --silent \ + --header "X-Vault-Token: ${vault_token}" \ + "${VAULT_ADDR%/}"/v1/"${PROJECT_SLUG}"/pact | \ + jq -r .data) PACT_BROKER_BASE_URL=$(echo "${pact_secrets}" | jq -r .pact_broker_base_url) PACT_BROKER_PASSWORD=$(echo "${pact_secrets}" | jq -r .pact_broker_password) @@ -17,5 +24,3 @@ if [ "${VAULT_ADDR}" != "" ]; then export PACT_BROKER_PASSWORD export PACT_BROKER_USERNAME fi - -docker-entrypoint.sh pact-broker "${@}" diff --git a/{{cookiecutter.project_dirname}}/scripts/ci_sentry.sh b/{{cookiecutter.project_dirname}}/scripts/ci_sentry.sh index 217b5acf..dc42d598 100755 --- a/{{cookiecutter.project_dirname}}/scripts/ci_sentry.sh +++ b/{{cookiecutter.project_dirname}}/scripts/ci_sentry.sh @@ -9,24 +9,24 @@ git config --global --add safe.directory "${PROJECT_DIR}" if [ "${VAULT_ADDR}" != "" ]; then apk add curl jq - vault_token=$(curl --silent --request POST --data "role=${VAULT_ROLE}" --data "jwt=${VAULT_ID_TOKEN}" "${VAULT_ADDR%/}"/v1/auth/gitlab-jwt/login | jq -r .auth.client_token) + vault_token=$(curl --silent --request POST \ + --data "role=${VAULT_ROLE}" \ + --data "jwt=${VAULT_ID_TOKEN}" \ + "${VAULT_ADDR%/}"/v1/auth/gitlab-jwt/login | \ + jq -r .auth.client_token) + + vault_base_secrets_addr="${VAULT_ADDR%/}/v1/${PROJECT_SLUG}/envs/${ENV_NAME}" + + SENTRY_AUTH_TOKEN=$(curl --silent \ + --header "X-Vault-Token: ${vault_token}" \ + "${vault_base_secrets_addr}/sentry" | \ + jq -r .data.sentry_auth_token) + + SENTRY_DSN=$(curl --silent \ + --header "X-Vault-Token: ${vault_token}" \ + "${vault_base_secrets_addr}/${SERVICE_SLUG}/sentry" | \ + jq -r .data.sentry_dsn) - SENTRY_AUTH_TOKEN=$(curl --silent --header "X-Vault-Token: ${vault_token}" "${VAULT_ADDR%/}"/v1/"${PROJECT_SLUG}"/envs/"${ENV_NAME}"/sentry | jq -r .data.sentry_auth_token) - SENTRY_DSN=$(curl --silent --header "X-Vault-Token: ${vault_token}" "${VAULT_ADDR%/}"/v1/"${PROJECT_SLUG}"/envs/"${ENV_NAME}"/"${SERVICE_SLUG}"/sentry | jq -r .data.sentry_dsn) export SENTRY_AUTH_TOKEN export SENTRY_DSN fi - -case "${1}" in - "release") - sentry-cli releases new "${VERSION_REF}" -p "${SENTRY_PROJECT_NAME}" --log-level=debug; - sentry-cli releases set-commits "${VERSION_REF}" --auto --ignore-missing; - sentry-cli releases finalize "${VERSION_REF}"; - ;; - "success") - sentry-cli releases deploys "${VERSION_REF}" new -e "${CI_ENVIRONMENT_NAME}" -t $((RELEASE_END-RELEASE_START)); - ;; - "failure") - sentry-cli send-event -m "Deploy to ${CI_ENVIRONMENT_NAME} failed."; - ;; -esac diff --git a/{{cookiecutter.project_dirname}}/scripts/ci_test_entrypoint.sh b/{{cookiecutter.project_dirname}}/scripts/ci_test_entrypoint.sh new file mode 100755 index 00000000..c27266ba --- /dev/null +++ b/{{cookiecutter.project_dirname}}/scripts/ci_test_entrypoint.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash + +set -euo pipefail + +if [ -n "${TEST_ENV_FILE:-}" ] && [ -f "${TEST_ENV_FILE}" ]; then + set -a && source "${TEST_ENV_FILE}" && set +a +fi + +uv sync --frozen --group remote --group test + +./scripts/entrypoint.sh "$@" diff --git a/{{cookiecutter.project_dirname}}/scripts/coverage.sh b/{{cookiecutter.project_dirname}}/scripts/coverage.sh index 347314c8..3dd9da90 100755 --- a/{{cookiecutter.project_dirname}}/scripts/coverage.sh +++ b/{{cookiecutter.project_dirname}}/scripts/coverage.sh @@ -2,4 +2,4 @@ set -euo pipefail -python3 -m coverage run manage.py test --configuration=Testing --noinput --parallel --shuffle --buffer +uv run coverage run -m pytest --no-migrations "$@" diff --git a/{{cookiecutter.project_dirname}}/scripts/deploy.sh b/{{cookiecutter.project_dirname}}/scripts/deploy.sh deleted file mode 100755 index 3f391600..00000000 --- a/{{cookiecutter.project_dirname}}/scripts/deploy.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/usr/bin/env sh - -set -e - -# init.sh must be sourced to let it export env vars -. "${PROJECT_DIR}"/scripts/deploy/init.sh - -sh "${PROJECT_DIR}"/scripts/deploy/terraform.sh validate - -sh "${PROJECT_DIR}"/scripts/deploy/terraform.sh plan-json - -sh "${PROJECT_DIR}"/scripts/deploy/terraform.sh apply -auto-approve diff --git a/{{cookiecutter.project_dirname}}/scripts/deploy/gitlab.sh b/{{cookiecutter.project_dirname}}/scripts/deploy/gitlab.sh deleted file mode 100755 index 69a45af0..00000000 --- a/{{cookiecutter.project_dirname}}/scripts/deploy/gitlab.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env sh - -set -e - -# If TF_USERNAME is unset then default to GITLAB_USER_LOGIN -TF_USERNAME="${TF_USERNAME:-${GITLAB_USER_LOGIN}}" -# If TF_PASSWORD is unset then default to gitlab-ci-token/CI_JOB_TOKEN -if [ -z "${TF_PASSWORD}" ]; then -TF_USERNAME="gitlab-ci-token" -TF_PASSWORD="${CI_JOB_TOKEN}" -fi -# If TF_ADDRESS is unset but TF_STATE_NAME is provided, then default to GitLab backend in current project -if [ -n "${TF_STATE_NAME}" ]; then -TF_ADDRESS="${TF_ADDRESS:-${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}}" -fi -# Set variables for the HTTP backend to default to TF_* values -export TF_HTTP_ADDRESS="${TF_HTTP_ADDRESS:-${TF_ADDRESS}}" -export TF_HTTP_LOCK_ADDRESS="${TF_HTTP_LOCK_ADDRESS:-${TF_ADDRESS}/lock}" -export TF_HTTP_LOCK_METHOD="${TF_HTTP_LOCK_METHOD:-POST}" -export TF_HTTP_UNLOCK_ADDRESS="${TF_HTTP_UNLOCK_ADDRESS:-${TF_ADDRESS}/lock}" -export TF_HTTP_UNLOCK_METHOD="${TF_HTTP_UNLOCK_METHOD:-DELETE}" -export TF_HTTP_USERNAME="${TF_HTTP_USERNAME:-${TF_USERNAME}}" -export TF_HTTP_PASSWORD="${TF_HTTP_PASSWORD:-${TF_PASSWORD}}" -export TF_HTTP_RETRY_WAIT_MIN="${TF_HTTP_RETRY_WAIT_MIN:-5}" -# Expose Gitlab specific variables to terraform since no -tf-var is available -# Usable in the .tf file as variable "CI_JOB_ID" { type = string } etc -export TF_VAR_CI_JOB_ID="${TF_VAR_CI_JOB_ID:-${CI_JOB_ID}}" -export TF_VAR_CI_COMMIT_SHA="${TF_VAR_CI_COMMIT_SHA:-${CI_COMMIT_SHA}}" -export TF_VAR_CI_JOB_STAGE="${TF_VAR_CI_JOB_STAGE:-${CI_JOB_STAGE}}" -export TF_VAR_CI_PROJECT_ID="${TF_VAR_CI_PROJECT_ID:-${CI_PROJECT_ID}}" -export TF_VAR_CI_PROJECT_NAME="${TF_VAR_CI_PROJECT_NAME:-${CI_PROJECT_NAME}}" -export TF_VAR_CI_PROJECT_NAMESPACE="${TF_VAR_CI_PROJECT_NAMESPACE:-${CI_PROJECT_NAMESPACE}}" -export TF_VAR_CI_PROJECT_PATH="${TF_VAR_CI_PROJECT_PATH:-${CI_PROJECT_PATH}}" -export TF_VAR_CI_PROJECT_URL="${TF_VAR_CI_PROJECT_URL:-${CI_PROJECT_URL}}" diff --git a/{{cookiecutter.project_dirname}}/scripts/deploy/init.sh b/{{cookiecutter.project_dirname}}/scripts/deploy/init.sh deleted file mode 100755 index c5902c69..00000000 --- a/{{cookiecutter.project_dirname}}/scripts/deploy/init.sh +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env sh - -set -e - -export TF_VAR_env_slug="${ENV_SLUG}" -export TF_VAR_project_slug="${PROJECT_SLUG}" -export TF_VAR_stack_slug="${STACK_SLUG}" - -terraform_cli_args="-var-file=${TERRAFORM_VARS_DIR%/}/.tfvars" - -if [ "${TERRAFORM_EXTRA_VAR_FILE}" != "" ]; then - extra_var_file="${TERRAFORM_VARS_DIR%/}/${TERRAFORM_EXTRA_VAR_FILE}" - touch "${extra_var_file}" - terraform_cli_args="${terraform_cli_args} -var-file=${extra_var_file}" -fi - -if [ "${VAULT_ADDR}" != "" ]; then - . "${PROJECT_DIR}"/scripts/deploy/vault.sh - terraform_cli_args="${terraform_cli_args} -var-file=${TERRAFORM_VARS_DIR%/}/vault-secrets.tfvars.json" -fi - -export TF_CLI_ARGS_destroy="${terraform_cli_args}" -export TF_CLI_ARGS_plan="${terraform_cli_args}" - -case "${TERRAFORM_BACKEND}" in - "gitlab") - . "${PROJECT_DIR}"/scripts/deploy/gitlab.sh - ;; - "terraform-cloud") - . "${PROJECT_DIR}"/scripts/deploy/terraform-cloud.sh - ;; -esac diff --git a/{{cookiecutter.project_dirname}}/scripts/deploy/terraform-cloud.sh b/{{cookiecutter.project_dirname}}/scripts/deploy/terraform-cloud.sh deleted file mode 100755 index b01a928f..00000000 --- a/{{cookiecutter.project_dirname}}/scripts/deploy/terraform-cloud.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/usr/bin/env sh - -set -e - -export TF_CLI_CONFIG_FILE="${TF_ROOT}/cloud.tfc" -cat << EOF > "${TF_CLI_CONFIG_FILE}" -{ - "credentials": { - "app.terraform.io": { - "token": "${TFC_TOKEN}" - } - } -} -EOF diff --git a/{{cookiecutter.project_dirname}}/scripts/deploy/terraform.sh b/{{cookiecutter.project_dirname}}/scripts/deploy/terraform.sh deleted file mode 100755 index 43ac5c1e..00000000 --- a/{{cookiecutter.project_dirname}}/scripts/deploy/terraform.sh +++ /dev/null @@ -1,69 +0,0 @@ -#!/usr/bin/env sh - -set -e - -if [ "${DEBUG_OUTPUT}" = "true" ]; then - set -x -fi - -plan_cache="plan.cache" -plan_json="plan.json" - -JQ_PLAN=' - ( - [.resource_changes[]?.change.actions?] | flatten - ) | { - "create":(map(select(.=="create")) | length), - "update":(map(select(.=="update")) | length), - "delete":(map(select(.=="delete")) | length) - } -' - -# Use terraform automation mode (will remove some verbose unneeded messages) -export TF_IN_AUTOMATION=true - -init() { - cd "${TF_ROOT}" - if [ "${TERRAFORM_BACKEND}" = "terraform-cloud" ]; then - terraform init "${@}" -input=false - else - terraform init "${@}" -input=false -reconfigure - fi -} - -case "${1}" in - "apply") - init - terraform "${@}" -input=false "${plan_cache}" - ;; - "destroy") - init - terraform "${@}" -auto-approve - ;; - "fmt") - terraform "${@}" -check -diff -recursive - ;; - "init") - # shift argument list β€žone to the leftβ€œ to not call 'terraform init init' - shift - init "${@}" - ;; - "plan") - init - terraform "${@}" -input=false -out="${plan_cache}" - ;; - "plan-json") - init - terraform plan -input=false -out="${plan_cache}" - terraform show -json "${plan_cache}" | \ - jq -r "${JQ_PLAN}" \ - > "${plan_json}" - ;; - "validate") - init -backend=false - terraform "${@}" - ;; - *) - terraform "${@}" - ;; -esac diff --git a/{{cookiecutter.project_dirname}}/scripts/deploy/vault.sh b/{{cookiecutter.project_dirname}}/scripts/deploy/vault.sh deleted file mode 100755 index 402802d8..00000000 --- a/{{cookiecutter.project_dirname}}/scripts/deploy/vault.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env sh - -set -e - -vault_token=$(curl --silent --request POST --data "role=${VAULT_ROLE}" --data "jwt=${VAULT_ID_TOKEN}" "${VAULT_ADDR%/}"/v1/auth/gitlab-jwt/login | jq -r .auth.client_token) - -secrets_data="{}" - -for secret_path in ${VAULT_SECRETS} -do - secret_data=$(curl --silent --header "X-Vault-Token: ${vault_token}" "${VAULT_ADDR%/}"/v1/"${PROJECT_SLUG}"/"${VAULT_SECRETS_PREFIX}"/"${secret_path}" | jq -r '.data // {}') || secret_data="{}" - secrets_data=$(echo "${secrets_data}" | jq --argjson new_data "${secret_data}" '. * $new_data') -done - -echo "${secrets_data}" > "${TERRAFORM_VARS_DIR%/}"/vault-secrets.tfvars.json - -if [ "${TERRAFORM_BACKEND}" = "terraform-cloud" ]; then - TFC_TOKEN=$(curl --silent --header "X-Vault-Token: ${vault_token}" "${VAULT_ADDR%/}"/v1/"${PROJECT_SLUG}"-tfc/creds/default | jq -r .data.token) - export TFC_TOKEN -fi diff --git a/{{cookiecutter.project_dirname}}/scripts/entrypoint.sh b/{{cookiecutter.project_dirname}}/scripts/entrypoint.sh index 40a9d053..0fb27d38 100755 --- a/{{cookiecutter.project_dirname}}/scripts/entrypoint.sh +++ b/{{cookiecutter.project_dirname}}/scripts/entrypoint.sh @@ -2,5 +2,6 @@ set -euo pipefail -python3 -m manage migrate --noinput -exec "${@}" +uv run -m manage migrate --noinput + +exec uv run "$@" diff --git a/{{cookiecutter.project_dirname}}/scripts/pact_verify.sh b/{{cookiecutter.project_dirname}}/scripts/pact_verify.sh index 5f67d469..7a1ca47a 100755 --- a/{{cookiecutter.project_dirname}}/scripts/pact_verify.sh +++ b/{{cookiecutter.project_dirname}}/scripts/pact_verify.sh @@ -2,6 +2,21 @@ set -euo pipefail -python3 -m pytest --dc=Testing --disable-warnings \ +if [ -n "${VAULT_ID_TOKEN:-}" ]; then + vault_token=$(curl --silent --request POST \ + --data "role=pact" \ + --data "jwt=${VAULT_ID_TOKEN}" \ + "${VAULT_ADDR%/}/v1/auth/gitlab-jwt/login" | \ + jq -r .auth.client_token) + + PACT_BROKER_URL=$(curl --silent \ + --header "X-Vault-Token: ${vault_token}" \ + "${VAULT_ADDR%/}/v1/${PROJECT_SLUG}/pact" | \ + jq -r .data.pact_broker_auth_url) + + export PACT_BROKER_URL +fi + +uv run pytest --disable-warnings \ --pact-provider-name="${PACT_PROVIDER_NAME}" \ "${@}" pacts/verify_pacts.py diff --git a/{{cookiecutter.project_dirname}}/scripts/report.sh b/{{cookiecutter.project_dirname}}/scripts/report.sh index 68ec1285..769febd9 100755 --- a/{{cookiecutter.project_dirname}}/scripts/report.sh +++ b/{{cookiecutter.project_dirname}}/scripts/report.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash -set -uo pipefail +set -euo pipefail -python3 -m coverage combine -python3 -m coverage html -python3 -m coverage report +uv run coverage combine +uv run coverage html +uv run coverage xml +uv run coverage report diff --git a/{{cookiecutter.project_dirname}}/scripts/test.sh b/{{cookiecutter.project_dirname}}/scripts/test.sh index 61d4b855..f5f06f34 100755 --- a/{{cookiecutter.project_dirname}}/scripts/test.sh +++ b/{{cookiecutter.project_dirname}}/scripts/test.sh @@ -1,8 +1,7 @@ #!/usr/bin/env bash -set -euo pipefail +set -uo pipefail ./scripts/check.sh -./scripts/coverage.sh -./scripts/behave.sh +./scripts/coverage.sh "$@" ./scripts/report.sh diff --git a/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/main.tf b/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/main.tf deleted file mode 100644 index 43f8144f..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/main.tf +++ /dev/null @@ -1,87 +0,0 @@ -locals { - environment_slug = { development = "dev", staging = "stage", production = "prod" }[lower(var.environment)] - - namespace = "${var.project_slug}-${local.environment_slug}" -} - -terraform { - required_providers { - digitalocean = { - source = "digitalocean/digitalocean" - version = "~> 2.36" - } - kubernetes = { - source = "hashicorp/kubernetes" - version = "~> 2.27" - } - random = { - source = "hashicorp/random" - version = "~> 3.6" - } - } -} - -/* Providers */ - -provider "digitalocean" { - token = var.digitalocean_token -} - -provider "kubernetes" { - host = data.digitalocean_kubernetes_cluster.main.endpoint - token = data.digitalocean_kubernetes_cluster.main.kube_config[0].token - cluster_ca_certificate = base64decode( - data.digitalocean_kubernetes_cluster.main.kube_config[0].cluster_ca_certificate - ) -} - -/* Data Sources */ - -data "digitalocean_kubernetes_cluster" "main" { - name = var.stack_slug == "main" ? "${var.project_slug}-k8s-cluster" : "${var.project_slug}-${var.stack_slug}-k8s-cluster" -} - -/* Deployment */ - -module "deployment" { - source = "../modules/kubernetes/deployment" - - environment = var.environment - environment_slug = local.environment_slug - - namespace = local.namespace - - project_slug = var.project_slug - project_url = var.project_url - - service_container_image = var.service_container_image - service_container_port = var.service_container_port - service_limits_cpu = var.service_limits_cpu - service_limits_memory = var.service_limits_memory - service_replicas = var.service_replicas - service_requests_cpu = var.service_requests_cpu - service_requests_memory = var.service_requests_memory - service_slug = var.service_slug - - media_storage = var.media_storage - - cache_url = var.cache_url - django_additional_allowed_hosts = var.django_additional_allowed_hosts - django_admins = var.django_admins - django_default_from_email = var.django_default_from_email - django_disable_server_side_cursors = var.django_disable_server_side_cursors - django_server_email = var.django_server_email - email_url = var.email_url - s3_access_id = var.s3_access_id - s3_bucket_name = var.s3_bucket_name - s3_file_overwrite = var.s3_file_overwrite - s3_host = var.s3_host - s3_region = var.s3_region - s3_secret_key = var.s3_secret_key - sentry_dsn = var.sentry_dsn - use_redis = var.use_redis - web_concurrency = var.web_concurrency - - extra_config_values = var.extra_config_values - extra_secret_values = var.extra_secret_values -} diff --git a/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/variables.tf b/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/variables.tf deleted file mode 100644 index 4fb3e3ab..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/variables.tf +++ /dev/null @@ -1,192 +0,0 @@ -variable "cache_url" { - type = string - description = "A Django cache URL override." - default = "" - sensitive = true -} - -variable "digitalocean_token" { - description = "The Digital Ocean access token." - type = string - sensitive = true -} - -variable "django_additional_allowed_hosts" { - type = string - description = "Additional entries of the DJANGO_ALLOWED_HOSTS environment variable ('127.0.0.1', 'localhost', the service slug and the project host are included by default)." - default = "" -} - -variable "django_admins" { - type = string - description = "The value of the DJANGO_ADMINS environment variable." - default = "" -} - -variable "django_configuration" { - type = string - description = "The value of the DJANGO_CONFIGURATION environment variable." - default = "Remote" -} - -variable "django_default_from_email" { - type = string - description = "The value of the DJANGO_DEFAULT_FROM_EMAIL environment variable." - default = "" -} - -variable "django_disable_server_side_cursors" { - type = string - description = "The value of the DJANGO_DISABLE_SERVER_SIDE_CURSORS environment variable." - default = "False" -} - -variable "django_server_email" { - type = string - description = "The value of the DJANGO_SERVER_EMAIL environment variable." - default = "" -} - -variable "email_url" { - type = string - description = "The email server connection url." - default = "" - sensitive = true -} - -variable "environment" { - type = string - description = "The name of the deploy environment, e.g. \"Production\"." -} - -variable "extra_config_values" { - type = map(string) - description = "Additional config map environment variables." - default = {} -} - -variable "extra_secret_values" { - type = map(string) - description = "Additional secret environment variables." - default = {} - sensitive = true -} - -variable "media_storage" { - description = "The media storage solution." - type = string -} - -variable "project_slug" { - description = "The project slug." - type = string -} - -variable "project_url" { - description = "The project url." - type = string -} - -variable "s3_access_id" { - description = "The S3 bucket access key ID." - type = string - default = "" - sensitive = true -} - -variable "s3_bucket_name" { - description = "The S3 bucket name." - type = string - default = "" -} - -variable "s3_file_overwrite" { - description = "The S3 bucket file overwriting setting." - type = string - default = "False" -} - -variable "s3_host" { - description = "The S3 bucket host." - type = string - default = "" -} - -variable "s3_region" { - description = "The S3 bucket region." - type = string - default = "" -} - -variable "s3_secret_key" { - description = "The S3 bucket secret access key." - type = string - default = "" - sensitive = true -} - -variable "sentry_dsn" { - description = "The Sentry project DSN." - type = string - default = "" - sensitive = true -} - -variable "service_container_image" { - description = "The service container image." - type = string -} - -variable "service_container_port" { - description = "The service container port." - type = string - default = "{{ cookiecutter.internal_service_port }}" -} - -variable "service_limits_cpu" { - description = "The service limits cpu value." - type = string -} - -variable "service_limits_memory" { - description = "The service limits memory value." - type = string -} - -variable "service_replicas" { - description = "The desired numbers of replicas to deploy." - type = number - default = 1 -} - -variable "service_requests_cpu" { - description = "The service requests cpu value." - type = string -} - -variable "service_requests_memory" { - description = "The service requests memory value." - type = string -} - -variable "service_slug" { - description = "The service slug." - type = string -} - -variable "stack_slug" { - description = "The slug of the stack where the service is deployed." - type = string -} - -variable "use_redis" { - description = "Tell if a Redis service is used." - type = bool - default = false -} - -variable "web_concurrency" { - description = "The desired number of gunicorn workers." - type = string - default = "" -} diff --git "a/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/{% if cookiecutter.terraform_backend == \"gitlab\" %}backend.tf{% endif %}" "b/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/{% if cookiecutter.terraform_backend == \"gitlab\" %}backend.tf{% endif %}" deleted file mode 100644 index 4ca44e9b..00000000 --- "a/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/{% if cookiecutter.terraform_backend == \"gitlab\" %}backend.tf{% endif %}" +++ /dev/null @@ -1,4 +0,0 @@ -terraform { - backend "http" { - } -} diff --git "a/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/{% if cookiecutter.terraform_backend == \"terraform-cloud\" %}cloud.tf{% endif %}" "b/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/{% if cookiecutter.terraform_backend == \"terraform-cloud\" %}cloud.tf{% endif %}" deleted file mode 100644 index 3849a361..00000000 --- "a/{{cookiecutter.project_dirname}}/terraform/digitalocean-k8s/{% if cookiecutter.terraform_backend == \"terraform-cloud\" %}cloud.tf{% endif %}" +++ /dev/null @@ -1,9 +0,0 @@ -terraform { - cloud { - organization = "{{ cookiecutter.terraform_cloud_organization }}" - - workspaces { - tags = ["project:{{ cookiecutter.project_slug }}"] - } - } -} diff --git a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/cronjob/main.tf b/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/cronjob/main.tf deleted file mode 100644 index edfac35f..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/cronjob/main.tf +++ /dev/null @@ -1,76 +0,0 @@ -terraform { - required_providers { - kubernetes = { - source = "hashicorp/kubernetes" - version = "~> 2.27" - } - } -} - -/* Cron Job */ - -resource "kubernetes_cron_job_v1" "main" { - metadata { - name = var.name - namespace = var.namespace - } - - spec { - schedule = var.schedule - job_template { - metadata {} - spec { - template { - metadata {} - spec { - dynamic "volume" { - for_each = toset(var.media_persistent_volume_claim_name != "" ? [1] : []) - - content { - name = "media" - persistent_volume_claim { - claim_name = var.media_persistent_volume_claim_name - } - } - } - image_pull_secrets { - name = "regcred" - } - container { - name = "main" - image = var.container_image - command = var.container_command - dynamic "volume_mount" { - for_each = toset(var.media_persistent_volume_claim_name != "" ? [1] : []) - - content { - name = "media" - mount_path = var.media_mount_path - } - } - dynamic "env_from" { - for_each = toset(var.config_maps) - - content { - config_map_ref { - name = env_from.key - } - } - } - dynamic "env_from" { - for_each = toset(var.secrets) - - content { - secret_ref { - name = env_from.key - } - } - } - } - restart_policy = "OnFailure" - } - } - } - } - } -} diff --git a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/cronjob/variables.tf b/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/cronjob/variables.tf deleted file mode 100644 index ca4236b0..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/cronjob/variables.tf +++ /dev/null @@ -1,48 +0,0 @@ -variable "config_maps" { - description = "The CronJob ConfigMap names." - type = list(string) - default = [] -} - -variable "container_command" { - description = "The CronJob container command." - type = list(string) -} - -variable "container_image" { - description = "The CronJob container image." - type = string -} - -variable "media_mount_path" { - description = "The mount path of the media directory inside the container." - type = string - default = "/app/media" -} - -variable "media_persistent_volume_claim_name" { - description = "The media persistent volume claim name." - type = string - default = "" -} - -variable "name" { - type = string - description = "The CronJob name." -} - -variable "namespace" { - description = "The Kubernetes namespace." - type = string -} - -variable "schedule" { - description = "The CronJob schedule." - type = string -} - -variable "secrets" { - description = "The CronJob Secret names." - type = list(string) - default = [] -} diff --git a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/main.tf b/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/main.tf deleted file mode 100644 index eeed2c7e..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/main.tf +++ /dev/null @@ -1,207 +0,0 @@ -locals { - service_labels = { - component = var.service_slug - environment = var.environment - project = var.project_slug - terraform = "true" - } - - project_host = regexall("https?://([^/]+)", var.project_url)[0][0] - - django_allowed_hosts = join( - ",", - setunion( - split(",", coalesce(var.django_additional_allowed_hosts, "127.0.0.1,localhost")), - [local.project_host, var.service_slug] - ) - ) - - additional_secrets = var.use_redis ? ["database-url", "redis-url"] : ["database-url"] - - cache_url = var.cache_url != "" ? var.cache_url : var.use_redis ? "$(REDIS_URL)?key_prefix=${var.environment_slug}" : "" - - use_s3 = length(regexall("s3", var.media_storage)) > 0 -} - -terraform { - required_providers { - kubernetes = { - source = "hashicorp/kubernetes" - version = "~> 2.27" - } - random = { - source = "hashicorp/random" - version = "~> 3.6" - } - } -} - -/* Passwords */ - -resource "random_password" "django_secret_key" { - length = 50 -} - -/* Secrets */ - -resource "kubernetes_secret_v1" "main" { - - metadata { - name = "${var.service_slug}-env-vars" - namespace = var.namespace - } - - data = { for k, v in merge( - var.extra_secret_values, - { - DJANGO_SECRET_KEY = random_password.django_secret_key.result - EMAIL_URL = var.email_url - SENTRY_DSN = var.sentry_dsn - }, - local.use_s3 ? { - AWS_ACCESS_KEY_ID = var.s3_access_id - AWS_SECRET_ACCESS_KEY = var.s3_secret_key - } : {} - ) : k => v if v != "" } -} - -/* Config Map */ - -resource "kubernetes_config_map_v1" "main" { - metadata { - name = "${var.service_slug}-env-vars" - namespace = var.namespace - } - - data = { for k, v in merge( - var.extra_config_values, - { - DJANGO_ADMINS = var.django_admins - DJANGO_ALLOWED_HOSTS = local.django_allowed_hosts - DJANGO_CONFIGURATION = "Remote" - DJANGO_CSRF_TRUSTED_ORIGINS = var.project_url - DJANGO_DEFAULT_FROM_EMAIL = var.django_default_from_email - DJANGO_DISABLE_SERVER_SIDE_CURSORS = var.django_disable_server_side_cursors - DJANGO_SERVER_EMAIL = var.django_server_email - DJANGO_SESSION_COOKIE_DOMAIN = local.project_host - INTERNAL_SERVICE_PORT = var.service_container_port - SENTRY_ENVIRONMENT = var.environment - WEB_CONCURRENCY = var.web_concurrency - }, - local.use_s3 ? { - AWS_S3_REGION_NAME = var.s3_region - DJANGO_AWS_LOCATION = "${var.environment_slug}/media" - DJANGO_AWS_S3_ENDPOINT_URL = var.media_storage == "digitalocean-s3" ? "https://${var.s3_region}.${var.s3_host}" : "" - DJANGO_AWS_S3_FILE_OVERWRITE = var.s3_file_overwrite - DJANGO_AWS_STORAGE_BUCKET_NAME = var.s3_bucket_name - } : {} - ) : k => v if v != "" } -} - -/* Deployment */ - -resource "kubernetes_deployment_v1" "main" { - metadata { - name = var.service_slug - namespace = var.namespace - annotations = { - "reloader.stakater.com/auto" = "true" - } - } - spec { - replicas = var.service_replicas - selector { - match_labels = local.service_labels - } - template { - metadata { - labels = local.service_labels - } - spec { - dynamic "volume" { - for_each = toset(var.media_persistent_volume_claim_name != "" ? [1] : []) - - content { - name = "media" - persistent_volume_claim { - claim_name = var.media_persistent_volume_claim_name - } - } - } - image_pull_secrets { - name = "regcred" - } - container { - image = var.service_container_image - name = var.service_slug - resources { - limits = { - cpu = var.service_limits_cpu - memory = var.service_limits_memory - } - requests = { - cpu = var.service_requests_cpu - memory = var.service_requests_memory - } - } - port { - container_port = var.service_container_port - } - dynamic "volume_mount" { - for_each = toset(var.media_persistent_volume_claim_name != "" ? [1] : []) - - content { - name = "media" - mount_path = var.media_mount_path - } - } - env_from { - config_map_ref { - name = kubernetes_config_map_v1.main.metadata[0].name - } - } - env_from { - secret_ref { - name = kubernetes_secret_v1.main.metadata[0].name - } - } - dynamic "env_from" { - for_each = toset(local.additional_secrets) - content { - secret_ref { - name = env_from.key - } - } - } - dynamic "env" { - for_each = toset(local.cache_url != "" ? [1] : []) - - content { - name = "CACHE_URL" - value = local.cache_url - } - } - } - } - } - } -} - -/* Cluster IP Service */ - -resource "kubernetes_service_v1" "cluster_ip" { - metadata { - name = var.service_slug - namespace = var.namespace - } - spec { - type = "ClusterIP" - selector = { - component = var.service_slug - } - port { - port = var.service_container_port - target_port = var.service_container_port - } - } -} diff --git a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/outputs.tf b/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/outputs.tf deleted file mode 100644 index 24f51b6b..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/outputs.tf +++ /dev/null @@ -1,9 +0,0 @@ -output "config_map_name" { - description = "The name of the Kubernetes ConfigMap associated with the Deployment." - value = kubernetes_config_map_v1.main.metadata[0].name -} - -output "secret_name" { - description = "The name of the Kubernetes Secret associated with the Deployment." - value = kubernetes_secret_v1.main.metadata[0].name -} diff --git a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/variables.tf b/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/variables.tf deleted file mode 100644 index 3e2f9f9d..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/modules/kubernetes/deployment/variables.tf +++ /dev/null @@ -1,196 +0,0 @@ -variable "cache_url" { - type = string - description = "A Django cache URL override." - default = "" - sensitive = true -} - -variable "django_additional_allowed_hosts" { - type = string - description = "Additional entries of the DJANGO_ALLOWED_HOSTS environment variable ('127.0.0.1', 'localhost', the service slug and the project host are included by default)." - default = "" -} - -variable "django_admins" { - type = string - description = "The value of the DJANGO_ADMINS environment variable." - default = "" -} - -variable "django_default_from_email" { - type = string - description = "The value of the DJANGO_DEFAULT_FROM_EMAIL environment variable." - default = "" -} - -variable "django_disable_server_side_cursors" { - type = string - description = "The value of the DJANGO_DISABLE_SERVER_SIDE_CURSORS environment variable." -} - -variable "django_server_email" { - type = string - description = "The value of the DJANGO_SERVER_EMAIL environment variable." - default = "" -} - -variable "email_url" { - type = string - description = "The email server connection url." - default = "" - sensitive = true -} - -variable "environment" { - type = string - description = "The deploy environment name, e.g. \"Production\"." -} - -variable "environment_slug" { - type = string - description = "The deploy environment slug, e.g. \"stage\"." -} - -variable "extra_config_values" { - type = map(string) - description = "Additional config map environment variables." - default = {} -} - -variable "extra_secret_values" { - type = map(string) - description = "Additional secret environment variables." - default = {} - sensitive = true -} - -variable "media_mount_path" { - description = "The mount path of the media directory inside the container." - type = string - default = "/app/media" -} - -variable "media_persistent_volume_claim_name" { - description = "The media persistent volume claim name." - type = string - default = "" -} - -variable "media_storage" { - description = "The media storage solution." - type = string -} - -variable "namespace" { - description = "The Kubernetes namespace." - type = string -} - -variable "project_slug" { - description = "The project slug." - type = string -} - -variable "project_url" { - description = "The project url." - type = string -} - -variable "s3_access_id" { - description = "The S3 bucket access key ID." - type = string - default = "" - sensitive = true -} - -variable "s3_bucket_name" { - description = "The S3 bucket name." - type = string - default = "" -} - -variable "s3_file_overwrite" { - description = "The S3 bucket file overwriting setting." - type = string - default = "False" -} - -variable "s3_host" { - description = "The S3 bucket host." - type = string - default = "" -} - -variable "s3_region" { - description = "The S3 bucket region." - type = string - default = "" -} - -variable "s3_secret_key" { - description = "The S3 bucket secret access key." - type = string - default = "" - sensitive = true -} - -variable "sentry_dsn" { - description = "The Sentry project DSN." - type = string - default = "" - sensitive = true -} - -variable "service_container_image" { - description = "The service container image." - type = string -} - -variable "service_container_port" { - description = "The service container port." - type = string - default = "{{ cookiecutter.internal_service_port }}" -} - -variable "service_limits_cpu" { - description = "The service limits cpu value." - type = string -} - -variable "service_limits_memory" { - description = "The service limits memory value." - type = string -} - -variable "service_replicas" { - description = "The desired numbers of replicas to deploy." - type = number - default = 1 -} - -variable "service_requests_cpu" { - description = "The service requests cpu value." - type = string -} - -variable "service_requests_memory" { - description = "The service requests memory value." - type = string -} - -variable "service_slug" { - description = "The service slug." - type = string -} - -variable "use_redis" { - description = "Tell if a Redis service is used." - type = bool - default = false -} - -variable "web_concurrency" { - description = "The desired number of gunicorn workers." - type = string - default = "" -} diff --git a/{{cookiecutter.project_dirname}}/terraform/other-k8s/main.tf b/{{cookiecutter.project_dirname}}/terraform/other-k8s/main.tf deleted file mode 100644 index d6977f1b..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/other-k8s/main.tf +++ /dev/null @@ -1,115 +0,0 @@ -locals { - environment_slug = { development = "dev", staging = "stage", production = "prod" }[lower(var.environment)] - - namespace = "${var.project_slug}-${local.environment_slug}" -} - -terraform { - required_providers { - kubernetes = { - source = "hashicorp/kubernetes" - version = "~> 2.27" - } - random = { - source = "hashicorp/random" - version = "~> 3.6" - } - } -} - -/* Providers */ - -provider "kubernetes" { - host = var.kubernetes_host - token = var.kubernetes_token - cluster_ca_certificate = base64decode(var.kubernetes_cluster_ca_certificate) -} - -/* Volumes */ - -resource "kubernetes_persistent_volume_v1" "media" { - count = var.media_storage == "local" ? 1 : 0 - - metadata { - name = "${local.namespace}-${var.service_slug}-media" - } - spec { - capacity = { - storage = var.media_persistent_volume_capacity - } - access_modes = ["ReadWriteOnce"] - persistent_volume_source { - host_path { - path = var.media_persistent_volume_host_path - } - } - } -} - -resource "kubernetes_persistent_volume_claim_v1" "media" { - count = var.media_storage == "local" ? 1 : 0 - - metadata { - name = "${var.service_slug}-media" - namespace = local.namespace - } - spec { - access_modes = ["ReadWriteOnce"] - resources { - requests = { - storage = coalesce( - var.media_persistent_volume_claim_capacity, - var.media_persistent_volume_capacity - ) - } - } - volume_name = kubernetes_persistent_volume_v1.media[0].metadata[0].name - } -} - -/* Deployment */ - -module "deployment" { - source = "../modules/kubernetes/deployment" - - environment = var.environment - environment_slug = local.environment_slug - - namespace = local.namespace - - project_slug = var.project_slug - project_url = var.project_url - - service_container_image = var.service_container_image - service_container_port = var.service_container_port - service_limits_cpu = var.service_limits_cpu - service_limits_memory = var.service_limits_memory - service_replicas = var.service_replicas - service_requests_cpu = var.service_requests_cpu - service_requests_memory = var.service_requests_memory - service_slug = var.service_slug - - media_storage = var.media_storage - - media_persistent_volume_claim_name = var.media_storage == "local" ? kubernetes_persistent_volume_claim_v1.media[0].metadata[0].name : "" - - cache_url = var.cache_url - django_additional_allowed_hosts = var.django_additional_allowed_hosts - django_admins = var.django_admins - django_default_from_email = var.django_default_from_email - django_disable_server_side_cursors = var.django_disable_server_side_cursors - django_server_email = var.django_server_email - email_url = var.email_url - s3_access_id = var.s3_access_id - s3_bucket_name = var.s3_bucket_name - s3_file_overwrite = var.s3_file_overwrite - s3_host = var.s3_host - s3_region = var.s3_region - s3_secret_key = var.s3_secret_key - sentry_dsn = var.sentry_dsn - use_redis = var.use_redis - web_concurrency = var.web_concurrency - - extra_config_values = var.extra_config_values - extra_secret_values = var.extra_secret_values -} diff --git a/{{cookiecutter.project_dirname}}/terraform/other-k8s/variables.tf b/{{cookiecutter.project_dirname}}/terraform/other-k8s/variables.tf deleted file mode 100644 index ac766408..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/other-k8s/variables.tf +++ /dev/null @@ -1,221 +0,0 @@ -variable "cache_url" { - type = string - description = "A Django cache URL override." - default = "" - sensitive = true -} - -variable "django_additional_allowed_hosts" { - type = string - description = "Additional entries of the DJANGO_ALLOWED_HOSTS environment variable ('127.0.0.1', 'localhost', the service slug and the project host are included by default)." - default = "" -} - -variable "django_admins" { - type = string - description = "The value of the DJANGO_ADMINS environment variable." - default = "" -} - -variable "django_configuration" { - type = string - description = "The value of the DJANGO_CONFIGURATION environment variable." - default = "Remote" -} - -variable "django_default_from_email" { - type = string - description = "The value of the DJANGO_DEFAULT_FROM_EMAIL environment variable." - default = "" -} - -variable "django_disable_server_side_cursors" { - type = string - description = "The value of the DJANGO_DISABLE_SERVER_SIDE_CURSORS environment variable." - default = "False" -} - -variable "django_server_email" { - type = string - description = "The value of the DJANGO_SERVER_EMAIL environment variable." - default = "" -} - -variable "email_url" { - type = string - description = "The email server connection url." - default = "" - sensitive = true -} - -variable "environment" { - type = string - description = "The name of the deploy environment, e.g. \"Production\"." -} - -variable "extra_config_values" { - type = map(string) - description = "Additional config map environment variables." - default = {} -} - -variable "extra_secret_values" { - type = map(string) - description = "Additional secret environment variables." - default = {} - sensitive = true -} - -variable "kubernetes_cluster_ca_certificate" { - description = "The base64 encoded Kubernetes CA certificate." - type = string - sensitive = true -} - -variable "kubernetes_host" { - description = "The Kubernetes host." - type = string -} - -variable "kubernetes_token" { - description = "A Kubernetes admin token." - type = string - sensitive = true -} - -variable "media_persistent_volume_capacity" { - description = "The media persistent volume capacity (e.g. 1Gi)." - type = string - default = "10Gi" -} - -variable "media_persistent_volume_claim_capacity" { - description = "The media persistent volume claim capacity (e.g. 1Gi)." - type = string - default = "" -} - -variable "media_persistent_volume_host_path" { - description = "The media persistent volume host path." - type = string - default = "" -} - -variable "media_storage" { - description = "The media storage solution." - type = string -} - -variable "project_slug" { - description = "The project slug." - type = string -} - -variable "project_url" { - description = "The project url." - type = string -} - -variable "s3_access_id" { - description = "The S3 bucket access key ID." - type = string - default = "" - sensitive = true -} - -variable "s3_bucket_name" { - description = "The S3 bucket name." - type = string - default = "" -} - -variable "s3_file_overwrite" { - description = "The S3 bucket file overwriting setting." - type = string - default = "False" -} - -variable "s3_host" { - description = "The S3 bucket host." - type = string - default = "" -} - -variable "s3_region" { - description = "The S3 bucket region." - type = string - default = "" -} - -variable "s3_secret_key" { - description = "The S3 bucket secret access key." - type = string - default = "" - sensitive = true -} - -variable "sentry_dsn" { - description = "The Sentry project DSN." - type = string - default = "" - sensitive = true -} - -variable "service_container_image" { - description = "The service container image." - type = string -} - -variable "service_container_port" { - description = "The service container port." - type = string - default = "" -} - -variable "service_limits_cpu" { - description = "The service limits cpu value." - type = string -} - -variable "service_limits_memory" { - description = "The service limits memory value." - type = string -} - -variable "service_replicas" { - description = "The desired numbers of replicas to deploy." - type = number - default = 1 -} - -variable "service_requests_cpu" { - description = "The service requests cpu value." - type = string -} - -variable "service_requests_memory" { - description = "The service requests memory value." - type = string -} - -variable "service_slug" { - description = "The service slug." - type = string -} - -variable "stack_slug" { - description = "The slug of the stack where the service is deployed." - type = string -} - -variable "use_redis" { - description = "Tell if a Redis service is used." - type = bool - default = false -} - -variable "web_concurrency" { - description = "The desired number of gunicorn workers." - type = string - default = "" -} diff --git "a/{{cookiecutter.project_dirname}}/terraform/other-k8s/{% if cookiecutter.terraform_backend == \"gitlab\" %}backend.tf{% endif %}" "b/{{cookiecutter.project_dirname}}/terraform/other-k8s/{% if cookiecutter.terraform_backend == \"gitlab\" %}backend.tf{% endif %}" deleted file mode 100644 index 4ca44e9b..00000000 --- "a/{{cookiecutter.project_dirname}}/terraform/other-k8s/{% if cookiecutter.terraform_backend == \"gitlab\" %}backend.tf{% endif %}" +++ /dev/null @@ -1,4 +0,0 @@ -terraform { - backend "http" { - } -} diff --git "a/{{cookiecutter.project_dirname}}/terraform/other-k8s/{% if cookiecutter.terraform_backend == \"terraform-cloud\" %}cloud.tf{% endif %}" "b/{{cookiecutter.project_dirname}}/terraform/other-k8s/{% if cookiecutter.terraform_backend == \"terraform-cloud\" %}cloud.tf{% endif %}" deleted file mode 100644 index 3849a361..00000000 --- "a/{{cookiecutter.project_dirname}}/terraform/other-k8s/{% if cookiecutter.terraform_backend == \"terraform-cloud\" %}cloud.tf{% endif %}" +++ /dev/null @@ -1,9 +0,0 @@ -terraform { - cloud { - organization = "{{ cookiecutter.terraform_cloud_organization }}" - - workspaces { - tags = ["project:{{ cookiecutter.project_slug }}"] - } - } -} diff --git a/{{cookiecutter.project_dirname}}/terraform/vars/.tfvars b/{{cookiecutter.project_dirname}}/terraform/vars/.tfvars deleted file mode 100644 index c21df49e..00000000 --- a/{{cookiecutter.project_dirname}}/terraform/vars/.tfvars +++ /dev/null @@ -1,15 +0,0 @@ -{% if "environment" in cookiecutter.tfvars %}{% for item in cookiecutter.tfvars.environment|sort %}{{ item }} -{% endfor %}{% endif %}# django_admins="" -# django_additional_allowed_hosts="" -# django_configuration="Remote" -# django_default_from_email="" -# django_disable_server_side_cursors="False" -# django_server_email="" -# s3_file_overwrite="False" -# service_container_port="{{ cookiecutter.internal_service_port }}" -service_limits_cpu="550m" -service_limits_memory="512Mi" -# service_replicas=1 -service_requests_cpu="25m" -service_requests_memory="115Mi" -# web_concurrency="" diff --git "a/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_dev\" in cookiecutter.tfvars %}dev.tfvars{% endif %}" "b/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_dev\" in cookiecutter.tfvars %}dev.tfvars{% endif %}" deleted file mode 100644 index e2140f3d..00000000 --- "a/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_dev\" in cookiecutter.tfvars %}dev.tfvars{% endif %}" +++ /dev/null @@ -1,2 +0,0 @@ -{% for item in cookiecutter.tfvars.environment_dev|sort %}{{ item }} -{% endfor %} diff --git "a/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_prod\" in cookiecutter.tfvars %}prod.tfvars{% endif %}" "b/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_prod\" in cookiecutter.tfvars %}prod.tfvars{% endif %}" deleted file mode 100644 index 6700bcd2..00000000 --- "a/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_prod\" in cookiecutter.tfvars %}prod.tfvars{% endif %}" +++ /dev/null @@ -1,2 +0,0 @@ -{% for item in cookiecutter.tfvars.environment_prod|sort %}{{ item }} -{% endfor %} diff --git "a/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_stage\" in cookiecutter.tfvars %}stage.tfvars{% endif %}" "b/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_stage\" in cookiecutter.tfvars %}stage.tfvars{% endif %}" deleted file mode 100644 index 11fc6807..00000000 --- "a/{{cookiecutter.project_dirname}}/terraform/vars/{% if \"environment_stage\" in cookiecutter.tfvars %}stage.tfvars{% endif %}" +++ /dev/null @@ -1,2 +0,0 @@ -{% for item in cookiecutter.tfvars.environment_stage|sort %}{{ item }} -{% endfor %}