Onboarding an enterprise client subscription
This is the ENTERPRISE path. An enterprise client gets its own Entra tenant and its own
subscription. It may hold admin rights there, and it can take the tenant over later. PartnerRe is
the first. An SME client instead gets its own subscription inside the shared MeshWeaver
Customers tenant and has no access to it: OnboardingSme.md. The fleet
statement behind both paths is Systemorph/Memex
docs/client-tenancy.md.
Two rules come before every step on this page. Both are maintainer directives of 2026-09-15.
- Everything is steered from memex.systemorph.com. It is the only control instance. The
client estate is changed only by a
Hosting/InstanceActionthere, approved in the mesh. memex dispatches the client config repository'saks-ops.ymlthrough thesystemorph-comGitHub App. There is no client control instance, no per-client GitHub App, no operator Job in the client cluster, and no Systemorph person with access to that cluster. - We host none of the client's data. The Postgres server and its databases, the backups, the
file shares behind the
content,attachments,usersanddataclaims, and the Key Vault all live in the client's subscription. None of it goes on a Systemorph resource: notmemexaks-pg, not theSystemorphvault, notmemexaks-clusteror its node resource group. Systemorph provides the image, the plugins and the registry, and stores nothing of the client's.
This is the guide for a Memex instance that runs in a customer's own Azure subscription rather than on the Systemorph cluster. It exists because the platform has one reference installation — the Systemorph estate, measured on 2026-09-15 and described at the end of this page — and every client subscription is that installation reproduced: same cluster shape, same identities, same vault naming, same registry, same chart. The reference is the model home; this page is the plan you build from.
Two audiences read it. The client's cloud team creates the resources in Part B and keeps them. Systemorph does Part C — the registry, the plugins, the plan — and, once the instance answers, signs in first. Neither side can finish alone, and the order below is the order in which the two halves meet.
An instance on the Systemorph cluster follows a shorter path, because its platform already
exists. That path is for Systemorph's own instances only, and it is not a path for a
client's data. The pearl trial went that way before the data rule existed, and its successor is
Pearl Technology's own subscription. The runbook for Systemorph's own instances lives in the config
repository (docs/new-deployment.md).
Its instance-level steps (the sign-in app, the instance key, the record) are the ones this page
reaches in Part B §7 and §8.
The shape you are building
client tenant / subscription Systemorph
┌───────────────────────────────────────────────────────┐ ┌────────────────────────────────┐
│ Resource group <rg> │ │ cr.meshweaver.cloud │
│ ┌── VNet ───────────────────────────────────────┐ │ │ portal + migration images │
│ │ AKS <prefix>-cluster (PRIVATE, 1.3x) │ │ │ plugin packages │
│ │ ├ ns <id>: memex-portal ×N ─ RWX /data … ──┼──┐ │ │ (pulled with the instance key)│
│ │ ├ app-routing-system (managed nginx) │ │ │ └────────────────────────────────┘
│ │ └ cert-manager (Let's Encrypt, HTTP-01) │ │ │ ┌────────────────────────────────┐
│ │ Postgres Flexible Server <prefix>-pg (private)│ │ │ │ memex.meshweaver.cloud │
│ └───────────────────────────────────────────────┘ │ │ │ the plugin registry: │
│ Azure Files (RWX shares) ◄─────────────────────────┘ │ │ /api/instances/register │
│ Key Vault <vault> (secret NAMES on the record) │ │ /api/plugins │
│ DNS zone <zone> A <host> → ingress IP │ └────────────────────────────────┘
│ Identities: <prefix>-portal-mi (workload identity), │
│ the Key Vault CSI addon identity, │ ┌────────────────────────────────┐
│ hosting-operator (the lane, OIDC) │ │ memex.systemorph.com │
│ In-VNet runner set (the operations lane) │ │ the ONLY control instance: │
└───────────────────────────────────────────────────────┘ │ records and approvals; it │
│ dispatches the client repo's │
│ aks-ops.yml as systemorph-com│
│ It stores no client data. │
└────────────────────────────────┘
Nine components, each with one job. The reference value is what the Systemorph estate runs today, and yours is what you decide in the next section. The reference column is the shape to copy. It is never a place for client data: every resource in the "Yours" column is created in the client's subscription.
| Component | Job | Reference (Systemorph) | Yours |
|---|---|---|---|
| Resource group + VNet | one blast radius, one address space | memex-aks-rg, 10.0.0.0/16, swedencentral |
<rg>, your CIDR, your region |
| AKS | runs the portal; private API server; OIDC issuer; workload identity; Key Vault CSI addon; managed nginx (app routing); Azure CNI overlay + Cilium | memexaks-cluster, Kubernetes 1.34, Standard tier |
<prefix>-cluster from the bicep |
| Postgres Flexible Server | the instance database, addressed by FQDN | memexaks-pg, PG 16, D8ds_v5, 128 GB, zone-redundant HA, 14-day backups, private access |
<prefix>-pg, PG 16, sized in §2 |
| Azure Files (RWX) | /data (framework cache, DataProtection keys), /mnt/content, /mnt/attachments, /mnt/users — shared by every replica |
azurefile-memex StorageClass, ZRS account |
the same class, your account |
| Key Vault | holds every secret by name; the record names them, never holds them | Systemorph (access policies) |
<vault> |
| DNS zone | the A record for the host, and the ACME challenge behind it | meshweaver.cloud in rg dns |
your zone, or a delegated subdomain |
| cert-manager | Let's Encrypt via HTTP-01 on the app-routing ingress class | letsencrypt-prod ClusterIssuer |
the same |
| Portal identity | user-assigned identity the pod runs as; one federated credential per namespace | memexaks-portal-mi, subject system:serviceaccount:<ns>:memex-portal-sa |
<prefix>-portal-mi |
| Registry access | images and plugins come from the fleet registry with the instance's own key | cr.meshweaver.cloud, pull Secret registry-pull |
the same key does both jobs |
What is deliberately not in the picture: a container registry of your own (optional mirror, §7),
a VPN gateway (not wanted: nobody runs kubectl against the cluster by hand), and a control
instance of the client's own. There is none, because memex.systemorph.com operates the estate (the
first decision below).
Decide first
Seven decisions change what gets created. Take them before §1; the recommendation is what the reference does unless a client constraint says otherwise.
| Decision | Options | Recommendation, and why |
|---|---|---|
| Tenancy | (a) enterprise: the client's own tenant and subscription; (b) SME: its own subscription in the shared MeshWeaver Customers tenant | (a) is this page, for a client with a directory of its own that may take it over. (b) is OnboardingSme.md. Either way the whole estate, and all of the client's data, lives in the client's subscription. |
| Operating model | not a choice: steered from memex.systemorph.com | The client estate is operated only from memex.systemorph.com. After an approval in the mesh, memex dispatches the client config repository's aks-ops.yml through systemorph-com, and the lane signs in to the client subscription as the client's hosting-operator identity for one job. A standalone model is not offered: no control instance of the client's own, no hand-run sample scripts, no operator Job. The part of this model that is not built yet is listed under Not built yet. |
| Where the images and plugins come from | (a) the fleet registry cr.meshweaver.cloud, pulled with the instance key; (b) a mirror in the client's own ACR |
(a). It is what every fleet instance does, the key is issued in §7 anyway, and the catalog verifies against the same registry. Mirror into an ACR only for an egress policy that forbids (a); crane copy as in the reference runbook, and grant the kubelet AcrPull. |
| Sign-in | (a) a multi-tenant Entra app in the client tenant, invitation-only; (b) single-tenant | (a). Invitation-only is what gates access (Features__Onboarding__InvitationOnly), not the audience; multi-tenant lets Systemorph support sign in with a Systemorph account when invited. The tenant id stays unset on a multi-tenant app. |
| DNS and host | (a) a host in the client's zone (memex.<client-domain>); (b) <id>.meshweaver.cloud in Systemorph's zone |
(a) when the client owns the brand; (b) is a Systemorph-side A record and a Systemorph-side certificate story. Either way the name must resolve publicly before TLS (HTTP-01). |
| Sizing | the envelope the record declares | Start from the reference customer envelope — portal requests 2 CPU / 4 GiB, limits 4 / 8 GiB; claims data 16 Gi, content 64 Gi, attachments 32 Gi, users 16 Gi — on a user pool that can hold the boot compile storm (a first boot recompiles every NodeType). The database runs in the cluster on the db pool (E4ds_v5 × 2, one per zone), one CloudNativePG release per instance — 4 GiB / 0.5 CPU and 32 Gi of zonal Premium SSD v2 per database instance as the starting envelope; the Flexible Server shape (D2ds_v5 for one instance) is the pre-2026-09-15 route. |
| Model access | (a) the client's own OpenRouter or provider key; (b) none at first | (a) if they have one, else (b). No provider configured is a working portal without chat, by design; the key is a vault object mapped on the record later (<prefix>OpenRouter-ApiKey). |
Also write down before starting: owner, update policy (Stable for a customer),
which plugins pre-install (Essentials is the platform baseline; a bare id is qualified
against the mount and fails closed), and data residency — the region above is the only place
the data lives, plus Postgres backups in the same region unless geo-backup is turned on. All of it
is in the client's subscription, and none of it is on a Systemorph resource.
Who does what
| Step | Client | Systemorph |
|---|---|---|
| Subscription, quotas, resource providers | ✅ | |
| §1–§6 platform: network, cluster, Postgres, vault, files, identities, DNS, TLS | ✅ | reviews the parameters |
| §7 registry: register the instance, receive the key | ✅ issues the key at the registry and grants the plan | |
| §7 put the key in the vault | ✅ (their vault) | |
| §8 the sign-in app + vault objects | ✅ (their tenant, their vault) | |
| §9 the record | ✅ reviews it | ✅ records it on memex.systemorph.com |
| §9 the first install | — | ✅ a Provision on memex.systemorph.com, approved in the mesh, run by the client repository's aks-ops.yml |
| Operations lane prerequisites | ✅ the hosting-operator identity and its federation, in its subscription |
✅ the config repository, the runner set, the org settings |
| §10 update policy, §11 observability | ✅ | advises |
| §12 verification | ✅ | ✅ first sign-in bootstraps the global admin — this is Systemorph's, then the client's admins are invited |
A client that wants Systemorph to run §1–§6 grants a Systemorph identity Owner or User Access Administrator on the resource group for the duration (the bicep authors role assignments) and removes it afterwards. Nothing in this guide needs a standing credential of Systemorph's in the client subscription.
Part A — before anything is created
Tooling. az ≥ 2.84 with az bicep ≥ 0.41, kubectl, helm ≥ 3.12, jq, curl. crane
only for the mirror option. Log in to the client subscription and tenant.
Quotas and providers. The cluster's VM family in the region (D-series v5), a public IP, and the
providers Microsoft.ContainerService, Microsoft.DBforPostgreSQL, Microsoft.KeyVault,
Microsoft.Storage, Microsoft.Network, Microsoft.ManagedIdentity. Quota requests are
asynchronous and can take a day: request them first.
az provider register -n Microsoft.ContainerService -n Microsoft.DBforPostgreSQL \
-n Microsoft.KeyVault -n Microsoft.Storage -n Microsoft.Network -n Microsoft.ManagedIdentity
az vm list-usage -l <region> --query "[?contains(name.value,'standardDSv5Family')]" -o table
The names, fixed once. Every later object derives from these; change them here or nowhere.
ID=<id> # instance id: 3–48 chars, lowercase letters, digits, single hyphens
# = namespace, database, helm release, registry instance id
PREFIX=$ID- # Key Vault object prefix
HOST=<host> # e.g. memex.client.example — must become a public A record
RG=<rg> # the resource group the bicep creates
NAMEPREFIX=<prefix> # bicep namePrefix: <prefix>-cluster, <prefix>-pg, <prefix>-portal-mi
VAULT=<vault> # the client's Key Vault
ZONE=<zone> # the DNS zone <host> lives in
Part B — what the client creates, in order
On the enterprise path the client's team may run these steps in its own subscription. When
Systemorph builds the estate instead, the same results come from the Infra deploy lane of the
client's config repository, dispatched as systemorph-com[bot]. A Systemorph person runs none of
the commands below by hand against the client's cluster.
1. Network, cluster, identities — the bicep
The platform's infrastructure is codified in the public sample:
deploy/aks/infra/main.bicep
in the MeshWeaver repository, a subscription-scoped deployment that creates the resource group,
the VNet with a delegated Postgres subnet, the private AKS cluster with OIDC issuer, workload
identity and the Key Vault CSI addon, the portal identity with one federated credential per
namespace, the Azure Files account with the named shares, the Postgres Flexible Server, and
(optionally) a VPN gateway and a backup storage account. Read its
README before running
it; the parameters that matter here:
| Parameter | Set to | Why |
|---|---|---|
namePrefix, resourceGroupName, location |
your names, your region | everything derives from them |
portalNamespaces |
["<id>"] |
one federated credential per entry, subject system:serviceaccount:<id>:memex-portal-sa. A namespace not in this list cannot run the portal with workload identity, and the failure is silent tag discovery |
sharedAcrLoginServer |
"" unless you mirror |
the reference uses a shared ACR the client does not have; images come from the fleet registry (§7) |
deployPostgresFlexible |
true; postgresSkuName per the sizing decision; postgresHighAvailability per SLA |
the record addresses it by FQDN |
deployContentFileShares |
true |
the four RWX shares; quotas are the bicep's *ShareQuotaGib parameters |
deployVpnGateway |
false unless the client wants interactive kubectl |
every step here uses az aks command invoke |
deployBackupStorage |
true if pgBackRest or the Hosting backup store is wanted; the Flexible Server keeps its own PITR either way |
|
systemNodeVmSize, minNodeCount, maxNodeCount |
the bicep defaults (D4s_v5, 3–6) for the system pool; add a user pool for the portal sized to the envelope | the reference runs its portals on a separate silos pool |
the db pool — REQUIRED |
Standard_E4ds_v5, 2 nodes, fixed (autoscaler off), zones 1 and 2, label workload=db, taint workload=db:NoSchedule |
each instance's PostgreSQL runs in the cluster as its own Helm release (a CloudNativePG Cluster, primary + standby one per zone) — MeshWeaver Doc/Architecture/InClusterDatabases. The pool must exist before the first Provision; the operator, the Barman Cloud plugin and the memex-db-premiumv2 StorageClass go on the cluster afterwards (MeshWeaver deploy/aks/manifests/db-platform/README.md) |
az bicep build --file infra/main.bicep --stdout > /dev/null # compiles clean
az deployment sub create -l <region> --template-file infra/main.bicep \
--parameters @infra/main.parameters.json --parameters postgresAdminPassword=<from a file, never argv>
az deployment sub show -n main --query properties.outputs # oidcIssuerUrl, clusterName, …
- You know it worked when
az aks show -g $RG -n $NAMEPREFIX-cluster --query "{private:apiServerAccessProfile.enablePrivateCluster,oidc:oidcIssuerProfile.enabled,wi:securityProfile.workloadIdentity.enabled,kv:addonProfiles.azureKeyvaultSecretsProvider.enabled,routing:ingressProfile.webAppRouting.enabled}"prints fivetrues. The reference prints exactly that. - 🚨 The OIDC issuer property is
issuerUrl, case-sensitive.--query oidcIssuerProfile.issuerURLprints nothing with exit 0, and a federated credential created from it has a blank issuer and fails every token exchange without an error naming it. Read it back.
2. Postgres — the server and the database
The bicep created the server; you create the database and confirm how it is reached.
az postgres flexible-server db create -g $RG -s $NAMEPREFIX-pg -d $ID
az postgres flexible-server show -g $RG -n $NAMEPREFIX-pg \
--query "{fqdn:fullyQualifiedDomainName,version:version,ha:highAvailability.mode,backupDays:backup.backupRetentionDays,public:network.publicNetworkAccess}"
- The instance addresses the server by FQDN, never a private IP — Azure moves a Flexible Server on scale, failover and maintenance; DNS follows, an IP does not.
- The admin password goes into the vault once (§3) as
<prefix>postgres-password; the connection string the portal reads is composed from it in §8 and stored under<prefix>db-connection. - Version 16, like the reference. The migration Job runs on first boot and the portal hard-gates on a migrated database.
3. Key Vault — and its access model
az keyvault create -g $RG -n $VAULT -l <region> --enable-rbac-authorization true
Two identities read it; nothing else does:
| Identity | Needs | How |
|---|---|---|
the Key Vault CSI addon identity (azurekeyvaultsecretsprovider-<cluster> in the node resource group) |
secrets get, list | Key Vault Secrets User on the vault (RBAC), or an access policy |
| whoever writes secrets (§7, §8) | secrets set | Key Vault Secrets Officer, or an access policy |
CSI=$(az aks show -g $RG -n $NAMEPREFIX-cluster --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv)
az role assignment create --assignee "$CSI" --role "Key Vault Secrets User" --scope "$(az keyvault show -n $VAULT --query id -o tsv)"
- 🚨 A vault on access policies ignores role assignments. The reference vault is on access
policies, and its operator identity carried a
Key Vault Secrets Officerrole for weeks while every write was refused, until a policy was added. Pick one model and grant in that model; readaz keyvault show --query properties.enableRbacAuthorizationbefore assuming. - The CSI identity's client id goes on the record as
keyVaultSecrets.identityClientId; the tenant id beside it.
4. Shared storage — the StorageClass and the claims
The portal runs as uid 1654 and keeps its framework cache, DataProtection keys and content on
ReadWriteMany shares that every replica mounts. A pod-local /data means every restart wipes
the compiled-assembly cache and every deploy becomes a compile storm.
az aks command invoke -g $RG -n $NAMEPREFIX-cluster \
--command "kubectl apply -f storageclass-azurefile.yaml" \
--file deploy/aks/manifests/storageclass-azurefile.yaml
The claims themselves are rendered by the chart from the record's volumes[] with
create: true (§9); nothing is applied by hand. The class is azurefile-memex: uid=1654,gid=1654,
mfsymlinks, nobrl, reclaim Retain — every option has a failure behind it, read the file's
header before changing one.
- You know it worked when, after §9,
kubectl -n $ID get deploy memex-portal-deployment -o jsonpath='{.spec.template.spec.volumes}'showsmemex-dataas apersistentVolumeClaim, neveremptyDir. Verify, do not assume.
5. Identities — the portal's, per namespace
The bicep created <prefix>-portal-mi and federated it to system:serviceaccount:<id>:memex-portal-sa.
Confirm the subject is exactly that string; a mismatch blocks tag discovery silently.
az identity federated-credential list -g $RG --identity-name $NAMEPREFIX-portal-mi \
--query "[].{name:name,subject:subject}" -o table
The client id of that identity is selfUpdate.azureClientId in the values (§9). No role is needed
on it for the fleet registry — the pull credential is the instance key (§7). Grant it AcrPull
only on a mirror ACR of your own.
6. DNS, ingress and TLS
The cluster's managed nginx (app routing) exposes one public IP. Point the host at it, then let cert-manager prove ownership over HTTP-01.
IP=$(az aks command invoke -g $RG -n $NAMEPREFIX-cluster \
--command "kubectl -n app-routing-system get svc nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}'" -o tsv --query logs)
az network dns record-set a add-record -g <dns-rg> -z $ZONE -n ${HOST%%.$ZONE} --ipv4-address "$IP" --ttl 300
dig +short $HOST # must answer the IP from OUTSIDE the VNet before the next step
cert-manager and the letsencrypt-prod ClusterIssuer are installed once per cluster by the
sample's tls.sh (Helm jetstack/cert-manager, solver http01 on ingress class
webapprouting.kubernetes.azure.com). The certificate is issued asynchronously: a Secret named
<id>-tls appears minutes after the Certificate, and until it does the ingress serves the default
certificate — which is what the first visitor sees. Wait for the Secret to carry a key pair.
- You know it worked when
curl -sS -o /dev/null -w "%{http_code} verify=%{ssl_verify_result}\n" https://$HOST/prints200 verify=0(or302to sign-in), from outside.
7. The registry key — issued by Systemorph, stored by the client
One key does two jobs: it is the plugin-catalog credential the Store presents to
memex.meshweaver.cloud, and the image-pull credential cr.meshweaver.cloud validates against
the same registry. It is issued, never generated: a random value boots an instance that
registers nothing and shows an empty Store with no error.
Systemorph registers the instance and hands over the key (returned once), or the client runs the open registration:
curl -sS -X POST https://memex.meshweaver.cloud/api/instances/register \
-H 'content-type: application/json' \
-d "{\"bootstrapKey\":\"\",\"instanceId\":\"$ID\",\"displayName\":\"<Name>\",\"homeUrl\":\"https://$HOST\"}" \
| jq -r .instanceKey > /tmp/$ID-key
head -c 4 /tmp/$ID-key; echo # mwi_
az keyvault secret set --vault-name $VAULT --name ${PREFIX}PluginCatalog-RegistryToken --file /tmp/$ID-key --output none
curl -sS -H "Authorization: Bearer $(cat /tmp/$ID-key)" https://memex.meshweaver.cloud/api/plugins | jq -r '.packages[].id' | head
rm /tmp/$ID-key
- Open registration enrols on the free plan. Systemorph raises the plan on the registry
(Instance grants ▸ Plan) — the instance cannot ask for it, and a package above the plan is
refused by name on the instance's
/health. - A taken id answers 409 and never says whose. A lost key is a rotation later, never a second registration.
- The pull Secret
registry-pull(kubernetes.io/dockerconfigjsonforcr.meshweaver.cloud, usernameinstance, password the key) is written in the namespace in §9 from this vault object.
8. The sign-in app and the vault objects
Four objects per instance, named by the rule <prefix><Section>-<Key> ↔ Section__Key
(Configuration → Secrets: the Key Vault naming rule). The record maps each
by name; the CSI driver mounts them at pod start; a declared object the vault does not hold
fails the whole mount and every pod stays ContainerCreating.
| Vault object | Value | Written by |
|---|---|---|
<prefix>Authentication-Microsoft-ClientSecret |
the Entra app's client secret | the client, below |
<prefix>PluginCatalog-RegistryToken |
the mwi_ key |
§7 |
<prefix>Ai-KeyProtection-MasterKey |
256 bits of random, minted once, never regenerated — it seals every stored provider credential | the client, below |
<prefix>db-connection |
the Npgsql connection string | the client, below |
Register the sign-in app in the client tenant:
APP_ID=$(az ad app create --display-name "<Name> Portal ($HOST)" \
--sign-in-audience AzureADMultipleOrgs \
--web-redirect-uris "https://$HOST/signin-microsoft" --query appId -o tsv)
az ad sp create --id "$APP_ID" --output none
umask 077
# az prints a minted secret on STDERR as well as in its output: always 2>/dev/null, file → vault, rm
az ad app credential reset --id "$APP_ID" --display-name "$ID" --years 1 \
--query password -o tsv > /tmp/$ID-secret 2>/dev/null
az keyvault secret set --vault-name $VAULT --name ${PREFIX}Authentication-Microsoft-ClientSecret \
--file /tmp/$ID-secret --output none
rm -f /tmp/$ID-secret
echo "$APP_ID" # → signIn.microsoftClientId on the record; the secret expires in a year: note the date
The master key and the connection string:
head -c 32 /dev/urandom | base64 | tr -d '\n' > /tmp/$ID-master
az keyvault secret set --vault-name $VAULT --name ${PREFIX}Ai-KeyProtection-MasterKey --file /tmp/$ID-master --output none
rm -f /tmp/$ID-master
PW=$(az keyvault secret show --vault-name $VAULT --name ${PREFIX}postgres-password --query value -o tsv)
printf 'Host=%s-pg.postgres.database.azure.com;Port=5432;Username=<admin>;Password=%s;Database=%s;SslMode=Require;Trust Server Certificate=true' \
"$NAMEPREFIX" "$PW" "$ID" > /tmp/$ID-db
unset PW
az keyvault secret set --vault-name $VAULT --name ${PREFIX}db-connection --file /tmp/$ID-db --output none
rm -f /tmp/$ID-db
- Never a secret on a command line (
--value), in a shell history, in a repository or in a record.--fileandrm. - You know it worked when
az keyvault secret list --vault-name $VAULT --query "[?starts_with(name,'$PREFIX')].name" -o tsvlists the four names, and everyshow --query idprints an id (never--query value). - A
Password=beside an Azure host is the correct shape: the portal selects Entra token auth only for an Azure host with no password.
9. The record, the values, the first install
The instance is a Hosting/Deployment record — identifiers and topology only, secrets by
name — and the values file is rendered from it. The record belongs to memex.systemorph.com's
Deployments space. Its repository field names the client's config repository, where
aks-ops.yml runs. Whether the file is authored in that repository and synced into memex, or held
by memex and carried signed in every dispatch, is open (Not built yet). The fields that decide
what the instance is:
{ "id": "<id>", "namespace": "Deployments", "nodeType": "Hosting/Deployment",
"content": { "$type": "DeploymentContent",
"host": "<host>", "namespace": "<id>", "cluster": "<prefix>-cluster",
"database": "<id>", "databaseServer": "<prefix>-pg", "databaseUsername": "<admin>",
"databaseConnectionSecret": "<prefix>db-connection",
"imageRepository": "cr.meshweaver.cloud/memex-portal-ai", "pinnedImageTag": "<a sealed tag>",
"imagePullSecret": "registry-pull",
"updatePolicy": "Stable", "owner": "<Client AG>", "status": "Provisioning",
"keyVault": "<vault>", "keyVaultSecretPrefix": "<prefix>",
"keyVaultSecrets": { "vaultName": "<vault>", "tenantId": "<client tenant>",
"identityClientId": "<CSI addon client id>", "name": "<id>-portal-keyvault",
"syncedSecret": "<id>-portal-keyvault", "volumeName": "kv-portal-keyvault",
"mountPath": "/mnt/kv-portal-keyvault",
"secrets": [
{ "key": "ConnectionStrings__memex", "vaultSecret": "<prefix>db-connection" },
{ "key": "Ai__KeyProtection__MasterKey", "vaultSecret": "<prefix>Ai-KeyProtection-MasterKey" },
{ "key": "Authentication__Microsoft__ClientSecret", "vaultSecret": "<prefix>Authentication-Microsoft-ClientSecret" },
{ "key": "PluginCatalog__RegistryToken", "vaultSecret": "<prefix>PluginCatalog-RegistryToken" },
{ "key": "PluginCatalog__Registries__0__Token", "vaultSecret": "<prefix>PluginCatalog-RegistryToken" } ] },
"signIn": { "provider": "Custom", "microsoftClientId": "<APP_ID>" },
"pluginRepos": [ { "name": "Plugins", "url": "https://memex.meshweaver.cloud" } ],
"preInstall": [ "Essentials" ],
"requiredModules": [ "MeshWeaver.Blazor.Radzen.dll", "MeshWeaver.Blazor.Analysis.dll",
"MeshWeaver.Blazor.EntityViews.dll", "MeshWeaver.Blazor.GoogleMaps.dll", "MeshWeaver.Speech.dll",
"MeshWeaver.Social.dll", "MeshWeaver.Blazor.Chat.dll", "MeshWeaver.Markdown.Collaboration.dll",
"MeshWeaver.AI.dll", "MeshWeaver.Mcp.dll" ],
"requiredModulesAuthoritative": true,
"extraPortalConfig": { "Features__Onboarding__InvitationOnly": "true" },
"replicas": 1,
"resources": { "requests": { "cpu": "2", "memory": "4Gi" }, "limits": { "cpu": "4", "memory": "8Gi" } },
"volumes": [
{ "name": "data", "claimName": "memex-data", "mountPath": "/data", "create": true, "size": "16Gi", "storageClass": "azurefile-memex" },
{ "name": "content", "claimName": "memex-content", "mountPath": "/mnt/content", "create": true, "size": "64Gi", "storageClass": "azurefile-memex" },
{ "name": "attachments", "claimName": "memex-attachments", "mountPath": "/mnt/attachments", "create": true, "size": "32Gi", "storageClass": "azurefile-memex" },
{ "name": "users", "claimName": "memex-users", "mountPath": "/mnt/users", "create": true, "size": "16Gi", "storageClass": "azurefile-memex" } ],
"ingress": { "className": "webapprouting.kubernetes.azure.com", "tlsSecret": "<id>-tls",
"sessionAffinity": { "enabled": true, "cookieName": "<ID>_AFFINITY" } },
"storage": { "backend": "Filesystem", "dataRoot": "/data", "contentPath": "/mnt/content",
"graphStorageType": "PostgreSql", "claudeCodeConfigDirRoot": "/mnt/users" },
"startupProbe": { "periodSeconds": 10, "timeoutSeconds": 5, "failureThreshold": 1080, "budgetSeconds": 10800 } } }
Four things in that record that are easy to get wrong, each with a live incident behind it:
pluginRepos[0].nameis the registry's name,Plugins, not the instance's. A barepreInstallid is qualified against it (Essentials→Plugins/Essentials) and the catalog fails closed: a mount named after the instance renders<id>/Essentials, matches nothing, and the portal comes up with an empty Store and no error.requiredModulesoverrides the image's list by index and never appends — so state the COMPLETE set and claim it withrequiredModulesAuthoritative: true. A list shorter than the image's leaves the image's tail required under names the record never wrote; an empty list restores the image's list in full. This page used to say "MeshWeaver.Mcp.dllgoes in slot 7 beside the image's 0–4", and that advice is RETIRED: arequiredModuleSlotsentry names an index in the IMAGE's list, which the record cannot read and which has grown from seven entries to nine underneath it — index 7 becameMeshWeaver.Markdown.Collaboration.dll, so the slot un-required the collaboration pack on the public instance and nothing reported it (Memex#131, then Memex#378). Since Plugins#2045HelmValues.Problemsasks core's two slot rules at render time (PositionalModuleSlotProblems: a slot on a record without the claim;ChartModuleSlotProblems: a slot the chart's literal-key block does not carry, aboveMaxChartRenderedRequiredModuleSlotor below 0), beside its own slot-inside-the-list check — so a Provision, Reconcile or composition review of such a record is REFUSED by name instead of rolling out green with a module required by nobody. The platform's own list is inMemex.Portal.Distributed/appsettings.json; copy it, append what the instance needs, and claim it. Platform reference:Doc/Architecture/RequiredModuleAuthority.Features__Onboarding__InvitationOnlyis enforced by nothing else. Omit it and the first render is an open portal, not an error. The first user on an empty user table bootstraps to global admin, so an invitation-only instance cannot lock itself out.claimNameis the field the chart reads. Without it/datais anemptyDirwiped on every restart;create: trueonly on a record whose claims do not exist yet.
The install is a Provision action on memex.systemorph.com, dry run first, then approved in
the mesh. memex dispatches the client repository's aks-ops.yml. Its mutate lane runs the
reference's hosting-deploy sequence on the in-VNet runner set: namespace, pull Secret, then
helm upgrade --install with the rendered values. Nobody runs deploy.sh by hand. Until the
items under Not built yet land, this path cannot complete for a second subscription, and the
hand-run path is not offered in its place. The migration Job runs on the upgrade; wait for Database migration completed. Version: N before
trusting the portal, and expect a cold-compile window of minutes on the first boot — do not cycle
pods, that restarts the work from cold.
10. Update policy and how a new build arrives
Set the policy to Stable in the portal (Admin ▸ Platform updates) as the first global admin. Then know what actually rolls the image:
- A client instance never patches its own cluster. The chart's
selfUpdate.canPatchstaysfalse, so nothing inside the client cluster holds the right to change it. - A new build reaches the instance as a
Rollaction on memex.systemorph.com. A roll to the tag already pinned on the record runs unattended on the read lane. A roll to any other tag needs an approval in the mesh. - Packages land inside the portal under their own update policy, and landing one does not touch the cluster.
Write this into the client's operating agreement rather than promising automatic updates.
11. Observability
The fleet watch sees a client instance only once remote observation exists (Not built yet). Until then the sample's
install-observability.sh puts Grafana, Loki, Promtail and Prometheus in monitoring, and the
OpenTelemetry collector DaemonSet archives pod logs to an Azure Files share. The reference
dashboards are bound to the reference cluster's datasource ids and are not a template; the
importer script is.
12. Verify — the checklist that says it is live
https://<host>/healthanswers200and its body readsHealthy, orDegradedonly for things you expect (no AI provider, a module lane not yet caught up).- Every volume on the portal Deployment is a
persistentVolumeClaim; none isemptyDirexcept dumps. /data/assembly-cacheaccumulates<Type>/vN-*.dlland survives a pod delete.- Sign-in works on the public host through the Entra app; the session survives a rollout
(DataProtection keys on
/data). - The plugin catalog shows one mount (
Plugins) and the packages the plan allows; install one and render its Tests area green. Green pods with an empty Store is the failure worth catching. curlfrom outside shows the real certificate, not the default one.- The record's
statusmoves toLive, by a reviewed commit.
Operations lane prerequisites
Each client config repository carries the operations lane, aks-ops.yml. It is a copy of the one
in Systemorph/Memex, adapted in three places: the actor gate, the cluster read from the record's
estate, and that estate's GitHub environments. It lives in the client repository because the
client identity's federation trusts that repository's environments and nobody else's.
Systemorph/PartnerRe.Memex is the first, and its
docs/aks-ops.md is the worked runbook.
| What | Where | Who |
|---|---|---|
aks-ops.yml and its five scripts, with the actor gate fixed to systemorph-com[bot] |
the client config repository | Systemorph |
environments aks-ops (read, unattended) and aks-ops-prod (mutate, mesh approval), deployable from main only |
the client config repository | Systemorph |
systemorph-com installed on the repository; the repository added to the org variable CONTROL_WEBHOOK_URL and the org secret CONTROL_WEBHOOK_SECRET |
the Systemorph organisation | Systemorph |
hosting-operator: a user-assigned identity federated to environment:aks-ops and environment:aks-ops-prod of that repository, in both subject shapes, and to nothing else |
the client subscription | whoever builds the estate |
| its rights: AKS Cluster Admin Role, Key Vault Secrets Officer, Contributor on the Postgres server, Managed Identity Contributor on the portal identity, DNS Zone Contributor on the zone | the client subscription | the same |
| an in-VNet runner set, registered to the Systemorph organisation in a runner group limited to that repository | the client cluster's network | Systemorph |
the record, with repository naming the client config repository and no operator block |
memex.systemorph.com | Systemorph |
What must remain. The actor gate. The mesh approval token, verified before any mutating step. The two federations and nothing more. The data plane in the client's subscription.
What must not exist:
- an operator Job or a control instance in the client estate;
- a per-client GitHub App;
- a federated credential for an in-cluster ServiceAccount or a branch;
- a person with a role on the client cluster;
- a laptop route of
az,kubectlorhelmby hand; - any client data on a Systemorph resource.
Egress the instance needs
A private cluster still needs outbound HTTPS to these, and a client egress policy is where a working instance most often goes dark without an error:
| Destination | For | When |
|---|---|---|
cr.meshweaver.cloud |
portal and migration images, plugin bundles | every pull, every module landing |
memex.meshweaver.cloud |
/api/instances/*, /api/plugins — registration, catalog, key validation |
boot, catalog refresh, key rotation |
acme-v02.api.letsencrypt.org and inbound :80 on the ingress IP |
HTTP-01 | certificate issue and renewal |
login.microsoftonline.com, graph.microsoft.com |
sign-in; mail only if a mail app is configured | every sign-in |
the model provider (openrouter.ai, or the client's own) |
chat and embeddings | only when a key is configured |
github.com |
GitSync, only if a Space is synced from a repository | optional |
github.com, *.actions.githubusercontent.com and the tool download hosts in aks-ops-tools.sh |
the in-VNet runner set of the operations lane | always, from the runner's network |
memex.systemorph.com |
the lane's signed aks-ops-result callback, which carries the step log and never a row, a file or a secret |
every operation |
*.postgres.database.azure.com (private DNS) |
the database | inside the VNet |
The reference estate, measured 2026-09-15
What the model home actually runs, so a deviation is a decision rather than an accident. It is the
shape of a client estate, never its location. Its Postgres, vault and file shares hold
Systemorph's own instances. A client instance never gets a database on memexaks-pg, an object in
the Systemorph vault, or a share in memexaks-cluster's node resource group.
| Component | Reference |
|---|---|
| Region, resource group | swedencentral, memex-aks-rg |
| AKS | memexaks-cluster, Kubernetes 1.34, Standard tier, private API server, OIDC + workload identity, Key Vault CSI addon, app routing, Azure CNI overlay; pools: system 2 × D4s_v3, silos 2–4 × D16s_v5 (the portals), two CI pools the client does not need |
| Portals on it | memex (control instance, 2 replicas), memex-cloud (public instance and registry host, 7 replicas), build, pearl (a customer trial that predates the data rule, replaced by Pearl Technology's own subscription, and not a model for client data) — one namespace, one database, one set of RWX claims each |
| Postgres | memexaks-pg, PG 16, GeneralPurpose D8ds_v5, 128 GB, zone-redundant HA, 14-day backups, private access, password auth |
| Key Vault | Systemorph, access policies; objects named <prefix><Section>-<Key>; per instance the four objects in §8 |
| Files | ZRS account, class azurefile-memex; ~1.1 TB across the portals |
| DNS | zones meshweaver.cloud, systemorph.com in rg dns; one shared ingress IP |
| Identities | memexaks-portal-mi (AcrPull on the shared ACR; federated per namespace to memex-portal-sa); hosting-operator (the control plane's lifecycle identity — Contributor on the Postgres server, DNS Zone Contributor, Key Vault Secrets Officer, Managed Identity Contributor on the portal identity, cluster admin; control instance only); the CSI addon identity (vault get, list) |
| Registry | images published to ACR meshweaver by CD; every non-registry instance pulls from cr.meshweaver.cloud with its instance key; plugins from the same registry |
| TLS | cert-manager, letsencrypt-prod, HTTP-01 on the app-routing class |
| Observability | Grafana, Loki, Prometheus in monitoring; the OpenTelemetry collector; the fleet watch samples every namespace |
Not built yet
memex.systemorph.com cannot yet operate a second subscription end to end. The platform changes it still needs:
- The executor. The core chart does not render
Hosting__Operator__ExecutororMaintainer, and memex.systemorph.com still runs the Job executor (Systemorph/Memex#313). - A second cluster and subscription per record. The control plane composes plans for its own cluster. A per-record cluster, subscription and identity is a platform change.
- Remote observation. The fleet board samples the control instance's own cluster only.
- Where the record lives. The record can be authored in the client repository and synced into memex, or held by memex and carried signed in every dispatch.
- The runner set. Each estate needs one, and which App registers it is open.
- One copy per repository. The lane is copied into each client repository until it becomes a reusable workflow (Systemorph/Memex#359).
What this page does not do
- It does not describe a standalone client estate. There is none: every client estate is operated from memex.systemorph.com.
- It does not size for load. The envelope above is the reference customer envelope; a client with many users or heavy content sizes the user pool, the Postgres tier and the shares from measurement.
- It does not cover a second instance in the same subscription. That is one more namespace, one
more database, one more federated credential in
portalNamespaces, one more record — the same four vault objects under a new prefix.