Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion .scripts/update_refs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,17 @@ if [[ "$CURRENT_BRANCH" == release-* ]]; then
| prepend "\t"
maybe_commit "chore(release): $MESSAGE"

# Replace 0.0.0-dev on lines marked `# release-ref: version`. These are
# versions that don't follow the `stackable0.0.0-dev` image tag pattern above,
# e.g. the operator version Argo CD installs in the argo-cd-git-ops stack.
MESSAGE="Update marked release versions to ${STACKABLE_RELEASE}.0"
echo "$MESSAGE"
find demos stacks -type f \
-exec grep --color=always -lE '0\.0\.0-dev\s+# release-ref: version' {} \; \
-exec sed -i -E "s#0\.0\.0-dev(\s+\# release-ref: version)#${STACKABLE_RELEASE}.0\1#" {} \; \
| prepend "\t"
maybe_commit "chore(release): $MESSAGE"

# Look for remaining references
echo "Checking files with older stackable release references which will be assumed to be intentional."
grep --color=always -ronE "stackable24\.3(\.[0-9]+)" | prepend "\t"
Expand All @@ -73,7 +84,7 @@ else
>&2 echo "WARNING: doesn't look like a release branch. Will not update stackableRelease versions in stacks and image references."
fi

MESSAGE="Replace githubusercontent references main->${CURRENT_BRANCH}"
MESSAGE="Replace main branch references with ${CURRENT_BRANCH}"
echo "$MESSAGE"
# Search for githubusercontent urls and replace the branch reference with a placeholder variable
# This is done just in case the branch has special regex characters (like `/`).
Expand All @@ -83,6 +94,14 @@ find demos stacks -type f \
-exec sed -i -E 's#(stackabletech/demos)/(refs/heads/)?main/#\1/\${UPDATE_BRANCH_REF}/#' {} \; \
| prepend "\t"

# Same for git refs that are not githubusercontent URLs, e.g. branches Argo CD
# or git-sync pull from at runtime. These lines are marked `# release-ref: branch`.
# shellcheck disable=SC2016 # We intentionally don't want to expand the variable.
find demos stacks -type f \
-exec grep --color=always -lE 'main\s+# release-ref: branch' {} \; \
-exec sed -i -E 's#main(\s+\# release-ref: branch)#${UPDATE_BRANCH_REF}\1#' {} \; \
| prepend "\t"

# Now, for all modified files, we can use envsubst
export UPDATE_BRANCH_REF="$CURRENT_BRANCH"
for MODIFIED_FILE in $(git diff --name-only); do
Expand All @@ -91,3 +110,12 @@ for MODIFIED_FILE in $(git diff --name-only); do
mv "$MODIFIED_FILE.replacements" "$MODIFIED_FILE"
done
maybe_commit "chore(release): $MESSAGE"

# A marked line that still points at main or 0.0.0-dev would leave a release
# depending on main at runtime, so treat it as an error.
if [[ "$CURRENT_BRANCH" == release-* ]]; then
if grep -rnE '(main|0\.0\.0-dev)\s+# release-ref:' demos stacks; then
>&2 echo "The lines above are marked as release references but were not updated. Exiting."
exit 3
fi
fi
41 changes: 40 additions & 1 deletion demos/argo-cd-git-ops/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ It provides the Sealed Secrets controller and a generated certificate that is au
```sh
kubeseal -n stackable-airflow --format=yaml < airflow-admin-credentials.yaml > sealed-airflow-admin-credentials.yaml
kubeseal -n stackable-airflow --format=yaml < airflow-postgresql-credentials.yaml > sealed-airflow-postgresql-credentials.yaml
kubeseal -n stackable-airflow --format=yaml < airflow-s3-connection.yaml > sealed-airflow-s3-connection.yaml
kubeseal -n garage --format=yaml < s3-credentials.yaml > sealed-s3-credentials.yaml
```

For "offline" usage you can first fetch the CA from the `sealed-secrets-controller` and use that for encryption when the demo / stack is not available.
Expand All @@ -34,7 +36,7 @@ Kubernetes rejects them as invalid label values, which breaks unsealing — labe

## Plaintext secrets used in this demo

The sealed secrets under `manifests/airflow/` were created from the following plaintext manifests.
The sealed secrets under `manifests/airflow/` and `manifests/garage/` were created from the following plaintext manifests.
These are demo-only credentials: the Sealed Secrets controller key for this stack is committed to this repository (`stacks/argo-cd-git-ops/secrets/sealed-secrets-key.yaml`), so the sealed secrets provide no confidentiality here — sealing only demonstrates the GitOps workflow.
Never use this pattern for real credentials.

Expand Down Expand Up @@ -69,4 +71,41 @@ stringData:
password: airflow
```

`s3-credentials.yaml` (namespace `garage`). Garage requires the `GK<24 hex characters>` key id format and a 64 hex character secret, so these are not free form values:

```yaml
---
apiVersion: v1
kind: Secret
metadata:
name: s3-credentials
type: Opaque
stringData:
accessKey: GK31c0ffee31c0ffee31c0ffee
secretKey: deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef
```

`airflow-s3-connection.yaml` (must use the key pair from `s3-credentials.yaml`):

```yaml
---
apiVersion: v1
kind: Secret
metadata:
name: airflow-s3-connection
type: Opaque
stringData:
airflow-s3-connection: aws://GK31c0ffee31c0ffee31c0ffee:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef@/?endpoint_url=https%3A%2F%2Fgarage.garage.svc.cluster.local%3A9000
```

TODO: We should probably create a script that seals all secrets and puts them in the respective stack / demo folders

## git-sync only fetches the DAG folder

A full clone of this repository needs several times the default memory of the git-sync containers.
git-sync therefore does a sparse partial clone of `dags/` only: `gitSyncConf` in `manifests/airflow/airflow.yaml`, with the sparse checkout file in `manifests/airflow/git-sync-sparse-checkout.yaml`.

- If the DAGs move, update both `gitFolder` in `airflow.yaml` and the sparse checkout file.
- git documents sparse checkout as experimental, so a newer git in the Airflow image may change how it behaves.
A breaking change fails silently: git-sync reports no errors, but the DAG folder is missing and Airflow shows no DAGs.
The fallback is a full clone, described in the comment above `gitSyncConf` in `airflow.yaml`: drop the sparse checkout configuration and give each git-sync container 512Mi of memory.
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: minio
name: garage
# Finalizer needed so resources created by the Application are deleted when the Application gets removed
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: minio
project: garage
destination:
server: https://kubernetes.default.svc
namespace: minio
namespace: garage
source:
repoURL: "{{ customGitUrl }}"
targetRevision: "{{ customGitBranch }}"
path: demos/argo-cd-git-ops/manifests/minio/
path: demos/argo-cd-git-ops/manifests/garage/
syncPolicy:
syncOptions:
- CreateNamespace=true
Expand Down
93 changes: 58 additions & 35 deletions demos/argo-cd-git-ops/manifests/airflow/airflow.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,20 +24,50 @@ spec:
# In terms of forking the repository, this should not be a problem since the DAGs are the same.
# You can still change this locally and point to the fork in case of any changes to DAGs.
#
# TODO(@maltesander): In a Stackable release, this branch should point to the release-YY.MM branch!
# The `release-ref` marker lets .scripts/update_refs.sh point this branch at
# the release branch when one is cut.
dagsGitSync:
- repo: https://github.com/stackabletech/demos/
branch: main
branch: main # release-ref: branch
gitFolder: "demos/argo-cd-git-ops/dags"
depth: 2
depth: 1
# Fetch and check out only the DAG folder (see git-sync-sparse-checkout.yaml).
# A full clone of this repository needs several times the git-sync
# containers' default memory, a sparse partial clone needs a fraction of it.
#
# git documents sparse checkout as experimental, so its behaviour may
# change with the git version in a future Airflow image. If it breaks, it
# does so silently: git-sync succeeds, but the DAG folder is missing and
# Airflow shows no DAGs. The fallback is a full clone: remove gitSyncConf,
# the git-sync-sparse volume and its mount, and git-sync-sparse-checkout.yaml,
# and give the git-sync containers (git-sync-0 and git-sync-0-init, in the
# podOverrides of every role including kubernetesExecutors) 512Mi of memory:
#
# - name: git-sync-0
# resources:
# limits:
# memory: "512Mi"
# requests:
# memory: "512Mi"
gitSyncConf:
--filter: blob:none
# See the mounted ConfigMap: git-sync-sparse-checkout
--sparse-checkout-file: /stackable/git-sync-sparse/sparse-checkout
Comment thread
adwk67 marked this conversation as resolved.
Resolved
volumes:
- name: minio-tls
# Mounted into the git-sync containers as well, which is how they find
# the sparse checkout file.
- name: git-sync-sparse
configMap:
name: git-sync-sparse-checkout
# Only the CA is needed, to verify the Garage S3 endpoint.
- name: s3-tls
ephemeral:
volumeClaimTemplate:
metadata:
annotations:
secrets.stackable.tech/class: tls
secrets.stackable.tech/scope: pod
secrets.stackable.tech/provision-parts: public
spec:
accessModes:
- ReadWriteOnce
Expand All @@ -46,44 +76,36 @@ spec:
storage: "1"
storageClassName: secrets.stackable.tech
volumeMounts:
- name: minio-tls
mountPath: /stackable/minio-tls
- name: git-sync-sparse
mountPath: /stackable/git-sync-sparse
- name: s3-tls
mountPath: /stackable/s3-tls
webservers:
roleConfig:
listenerClass: external-stable
envOverrides: &envOverrides
AIRFLOW_CONN_KUBERNETES_IN_CLUSTER: "kubernetes://?__extra__=%7B%22extra__kubernetes__in_cluster%22%3A+true%2C+%22extra__kubernetes__kube_config%22%3A+%22%22%2C+%22extra__kubernetes__kube_config_path%22%3A+%22%22%2C+%22extra__kubernetes__namespace%22%3A+%22%22%7D"
# Via sealed secrets and pod overrides, just kept for reference here
# AIRFLOW_CONN_MINIO: "aws://admin:adminadmin@/?endpoint_url=https%3A%2F%2Fminio.minio.svc.cluster.local%3A9000"
AWS_CA_BUNDLE: "/stackable/minio-tls/ca.crt"
# AIRFLOW_CONN_S3: "aws://GK31c0ffee31c0ffee31c0ffee:<secretKey>@/?endpoint_url=https%3A%2F%2Fgarage.garage.svc.cluster.local%3A9000"
AWS_CA_BUNDLE: "/stackable/s3-tls/ca.crt"
# boto3 has no region here otherwise, and Garage rejects requests signed
# for a different region than the one it is configured with.
AWS_DEFAULT_REGION: region-1
AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://demo/airflow-task-logs/
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: minio
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: s3
# 20 seconds to reload the DAGS folder (default: 300)
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "20"
podOverrides: &podOverrides
spec:
containers:
- name: airflow
env:
- name: AIRFLOW_CONN_MINIO
- name: AIRFLOW_CONN_S3
valueFrom:
secretKeyRef:
name: airflow-minio-connection
key: airflow-minio-connection
- name: git-sync-0
resources:
limits:
memory: "128Mi"
requests:
memory: "128Mi"
initContainers:
- name: git-sync-0-init
resources:
limits:
memory: "128Mi"
requests:
memory: "128Mi"
name: airflow-s3-connection
key: airflow-s3-connection
config:
gracefulShutdownTimeout: 30s
resources:
Expand All @@ -107,18 +129,11 @@ spec:
containers:
- name: base
env:
- name: AIRFLOW_CONN_MINIO
- name: AIRFLOW_CONN_S3
valueFrom:
secretKeyRef:
name: airflow-minio-connection
key: airflow-minio-connection
initContainers:
- name: git-sync-0-init
resources:
limits:
memory: "128Mi"
requests:
memory: "128Mi"
name: airflow-s3-connection
key: airflow-s3-connection
schedulers:
envOverrides: *envOverrides
podOverrides: *podOverrides
Expand All @@ -133,3 +148,11 @@ spec:
roleGroups:
default:
replicas: 1
# Without this role the DAG processor runs in the scheduler container, and the
# two together exceed the scheduler's memory limit.
dagProcessors:
envOverrides: *envOverrides
podOverrides: *podOverrides
roleGroups:
default:
replicas: 1
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
# Limits what git-sync checks out of this repository to the DAG folder. Used
# together with `--filter: blob:none` in airflow.yaml, so git-sync also only
# downloads the files it checks out.
#
# The git-sync image enables sparse checkout in cone mode, which requires every
# parent directory to be listed in this form. A single `/path/to/dags/` line
# checks out nothing but the top-level files.
#
# If the DAGs are moved, update both this file and `gitFolder` in airflow.yaml.
apiVersion: v1
kind: ConfigMap
metadata:
name: git-sync-sparse-checkout
labels:
stackable.tech/vendor: Stackable
stackable.tech/demo: argo-cd-git-ops
data:
sparse-checkout: |
/*
!/*/
/demos/
!/demos/*/
/demos/argo-cd-git-ops/
!/demos/argo-cd-git-ops/*/
/demos/argo-cd-git-ops/dags/

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: airflow-s3-connection
namespace: stackable-airflow
spec:
encryptedData:
airflow-s3-connection: AgCiOpa8CqtPyxatxCRHubdWYJGlrSPpstbD8wSgDkZFqFSaw+5bpKRFs0+CAnbq5ziCW4j3mSd/YMhaamqSCkQ/VWDH24VBSzlX/5tBcWNk93mMAtlZkBn2z1Uv8wMwX4sbdgdPnF9BzaelF24CBUpHTzBsFzPegBwRwAD4fqRA2jFueTSGfpIct+zLgUlAlUxn3UySSowE/12SCY+kkM/JLmF5jNLHwP54EIc8eBw0qVk9LF9zRrrhUeK9K7DZTmZO4wdaGBrQJgOteycMaMP0vRVvxK1+/MLURX5nVpvjKGWQdN8qtLA/5xezH2IUzhZgyn6iXJukvll+63RG1jnpDJAmYJtqktNz9fc/Y6lpn612a821zzS6DQS8dwqLbARhDROuAMcTiP3L2XAR48vR6ZJQoxPkGxkUWfUMOPFKIC5JeYQ5WQcQf74BRPYh5n497jif6PTabFPTCoIngodNGUd7dN0C4Si8XpUMXu8tMWFhzv4UeMaOw218oqGXuu2RTqYyJ2lsGM3poNBIX3GDSLOTBB9tblhmENX3yFs1SBj7g7j4Z8trFPj1ylM2UQsizI/rh+8165sGRJlHvHr7DwdHihLzLVUdKsPfwjiAWP0OU4ao/kzuzEI0gerpxltAp2Yix9o8KfzDj+xOG9GTDclhGc13DhTZMHjLSHZ1YCaR68JtCqceHGdG5OIjTVbHbfDU9VNLeabuQM7Ph2Q4tApAj+In5GFjArK4BK/2qaxoPY5+kc04DR1vWq/Ku3OUUMwBLvioFsWbtGRchs3LZceQTBHHiRNch4cct2oAn2AtQbvTjiFJC32zxfGpxV2Xx+NVuKs8OxM42UECa604zxs/0oOPKYoZBC6qbeQQS/uPh0ECJPfuKXU5QtYL6EIqteVDbPKX0HyQQK+D6TW0Y6aJ2iE=
template:
metadata:
creationTimestamp: null
name: airflow-s3-connection
namespace: stackable-airflow
labels:
stackable.tech/vendor: Stackable
stackable.tech/demo: argo-cd-git-ops
type: Opaque
Loading
Loading