Chapter 11: Packages and Escape Hatches

The derivation pipeline handles services, jobs, and models — workloads declared through platform CRDs. But not everything fits the pipeline. Third-party software (Redis, Prometheus, NGINX) ships as Helm charts. Legacy systems have their own deployment mechanisms. Edge cases require Kubernetes features the CRD doesn’t expose.

Every platform needs escape hatches. The decisions — when to use a new CRD vs. an escape hatch, how much of the security model to preserve, who maintains the bridge — are universal. The specific mechanisms shown here (Helm, LatticeMeshMember) are the reference implementation’s choices. The design decisions:

• When does a workload deserve a new CRD vs. an escape hatch? (This determines how much of the platform’s guarantees the workload gets.)
• How much of the platform’s security model should the escape hatch preserve? (All of it? Some? None?)
• Who maintains the bridge between the escape hatch and the platform? (The developer? The platform team? Automation?)

11.1 The Decision: New CRD or Escape Hatch?

This is the first question, and it should be answered before choosing a mechanism.

Signals that you need a new CRD:

• The workload has fundamentally different lifecycle semantics (batch jobs complete; services run forever).
• The workload has different scaling semantics (gang scheduling vs. horizontal autoscaling).
• The workload requires different resource types (VCJob instead of Deployment).
• Multiple teams need it — it’s a common pattern, not a one-off.

Signals that an escape hatch is sufficient:

• The workload is third-party software with its own lifecycle (Redis, Prometheus, NGINX).
• Only one team needs it.
• The workload’s deployment pattern doesn’t generalize to other teams.

The reference implementation has three CRD types (LatticeService, LatticeJob, LatticeModel) because three workload patterns are common enough to justify the derivation investment. Everything else — third-party databases, monitoring tools, message brokers — uses the escape hatch. Part VI covers the reasoning behind each CRD type.

The cost of getting this wrong in either direction: build a CRD for something only one team uses and you maintain a compiler nobody needs. Use an escape hatch for something every team needs and you lose the platform’s guarantees for a common workload.

11.2 Why Escape Hatches Are Inevitable

The categories of workload that don’t fit the derivation pipeline:

Third-party software. Databases, caches, message brokers, monitoring tools. They ship as Helm charts with their own lifecycle management. Rewriting them as platform CRDs is impractical — they have their own opinions about storage, networking, RBAC, and upgrades.

Platform infrastructure. The platform itself depends on software installed differently — Cilium, Istio, cert-manager, ESO, KEDA. These are prerequisites for the derivation pipeline. They can’t be derived by the pipeline they enable.

Edge cases. A service that needs a specific Kubernetes feature the CRD doesn’t expose (a custom volume type, a specific scheduling constraint, a non-standard port configuration). Extending the CRD for one team’s edge case bloats the API for everyone.

The design tension: every escape hatch is a hole in the platform’s guarantees. A Helm chart installed outside the pipeline doesn’t get automatic network policies, secret resolution, observability wiring, or quota enforcement.

11.3 The LatticePackage CRD

The primary escape hatch for third-party software: a platform CRD that orchestrates Helm chart installation.

apiVersion: lattice.dev/v1alpha1
kind: LatticePackage
metadata:
  name: redis
  namespace: data
spec:
  chart:
    repo: oci://registry.example.com/charts
    name: redis
    version: 18.4.0
  values:
    replicas: 3
    auth:
      enabled: true
      password: "${resources.redis.auth-token}"
  resources:
    redis:
      type: secret
      params:
        keys: [auth-token]

What the platform provides:

• Lifecycle management. Install, upgrade, rollback, and uninstall are reconciled by the platform’s controller — not by a human running helm install.
• Values interpolation. The LatticePackage spec supports a resources block — the same resource system as LatticeService’s workload.resources. Secret references in values (${resources.redis.auth-token}) are resolved through the same routing paths from Chapter 9. The chart receives the secret value; the developer references it the same way they would in a service spec. This means packages get the platform’s secret backend abstraction and Cedar AccessSecret authorization even though they don’t get the full derivation pipeline.
• Dependency ordering. Packages can declare dependencies on other packages. The platform installs them in the correct order and waits for health before proceeding.
• Garbage collection. If a LatticePackage is deleted, the platform uninstalls the Helm release and cleans up orphaned resources.

What the platform does NOT provide directly:

• Compile-time authorization for the chart’s contents (the chart installs whatever resources its templates produce).
• Observability wiring (the chart may or may not include ServiceMonitor resources).
• Image verification (the chart pulls whatever images it references — they don’t go through the TrustPolicy/cosign pipeline).

A package deployment, step by step. An operator applies the Redis LatticePackage above.

1. t=0. Admission webhook validates: chart repo uses OCI protocol, namespace is not a system namespace, version is pinned.
2. t=1s. The package controller resolves the ${resources.redis.auth-token} reference. Cedar AccessSecret evaluates: is this namespace permitted to access this secret? If denied, the package fails with a status condition — same as a service.
3. t=3s. ESO syncs the secret from the backend. The controller renders the Helm values with the resolved secret value substituted.
4. t=5s. The controller runs helm install with the rendered values. Redis pods start.
5. t=15s. The controller checks the Helm release health. Redis is running. Status: phase: Ready.

The operator didn’t run helm install manually. The secret was resolved through the platform’s backend. Cedar authorized the access. If the LatticePackage is deleted, the controller runs helm uninstall and cleans up.

What the operator must do separately: create a LatticeMeshMember (Section 11.4) to bring Redis into the bilateral agreement model. Without it, Redis has no network policies — any pod in the cluster can reach it (assuming the mesh allows traffic to non-enrolled pods, which depends on the cluster’s default-deny configuration).

11.4 Bridging Packages into the Mesh: LatticeMeshMember

The most significant gap for packaged workloads is network security. Helm-installed pods aren’t derived through the bilateral agreement model — they don’t get automatic CiliumNetworkPolicies or AuthorizationPolicies.

The LatticeMeshMember CRD bridges this gap. It lets operators declare mesh participation for workloads that weren’t derived by the platform:

apiVersion: lattice.dev/v1alpha1
kind: LatticeMeshMember
metadata:
  name: redis
  namespace: data
spec:
  target:
    selector:
      app: redis
  ports:
    - port: 6379
      name: redis
      peerAuth: Strict
  allowedCallers:
    - name: checkout
      namespace: commerce
    - name: inventory
      namespace: commerce
  dependencies: []

This tells the platform: “Redis exists, it listens on 6379 with mTLS, and it accepts traffic from checkout and inventory.” The mesh-member controller generates CiliumNetworkPolicies and AuthorizationPolicies for Redis — the same policies it generates for derived services. The bilateral agreement model applies: if checkout declares redis as an outbound dependency in its LatticeService spec, and the LatticeMeshMember declares checkout as an allowed caller, the policies are compiled on both sides.

The MeshMember concept is the bridge between the escape hatch and the security model. Packages don’t get the full derivation pipeline, but they can participate in the mesh and the bilateral agreement model. This narrows the security gap significantly.

LatticeMeshMember as a migration tool. If you’re migrating from Helm to the platform incrementally — one service at a time — LatticeMeshMember is how existing Helm-deployed services participate in bilateral agreements with newly platform-derived services. Migrate Service A to a LatticeService. Service B is still on Helm. Create a LatticeMeshMember for Service B that declares its ports and callers. Service A’s bilateral agreements work against Service B’s MeshMember. When Service B eventually migrates to a LatticeService, delete the MeshMember — the derivation pipeline takes over. This lets you migrate service-by-service without a big-bang cutover, and every service gets network policy coverage from the moment it has a MeshMember, whether it’s derived or manual.

11.5 Validation and Policy

What the platform can enforce on packages even without full derivation:

• Repository allowlisting. Only charts from approved registries. Prevents teams from pulling arbitrary charts from the internet.
• Namespace restrictions. Packages cannot target system namespaces (kube-system, istio-system, lattice-system). Only the platform team installs infrastructure there.
• Version pinning. Charts must specify an exact version, not latest or a range.
• HTTPS/OCI only. Chart repositories must use HTTPS or OCI protocols.

These are enforced by the LatticePackage admission webhook.

11.6 MeshMember Maintenance

The LatticeMeshMember is a manual bridge. Someone has to create it, and someone has to keep it in sync with the Helm chart it describes.

Who maintains it? Typically the team that installed the package. They know what ports Redis exposes and which services need to call it. The platform team may maintain MeshMembers for infrastructure packages (monitoring, logging) that multiple teams consume.

What drifts? The Helm chart version bumps and adds a new port (Redis adds a metrics exporter on 9121). The MeshMember still declares only port 6379. The new port has no network policies — it’s either unreachable (default-deny) or unprotected (if the mesh allows non-declared ports). The operator must update the MeshMember when the chart’s port surface changes.

How to detect drift. The platform can’t automatically detect that a package’s actual ports differ from its MeshMember’s declared ports — the package controller doesn’t inspect the chart’s templates. The compliance controller (Chapter 14) can probe: “are there pods in this namespace with open ports that no MeshMember declares?” This catches undeclared ports but requires the compliance scan to run.

The trade-off. MeshMembers are manual overhead. For a fleet with 5 packages, this is manageable. For a fleet with 50 packages, each with version bumps and port changes, MeshMember maintenance becomes a significant operational cost. This is one of the factors in the “new CRD vs escape hatch” decision from Section 11.1: if a package is common enough that maintaining its MeshMember across many clusters is expensive, it may be worth building a CRD that derives the mesh integration automatically.

11.7 What Goes Wrong in Practice

Scenario: package deployed without a MeshMember. An operator installs Redis via LatticePackage. They forget to create the LatticeMeshMember. Redis is running — pods are healthy, the Helm release is Ready.

In a default-deny cluster, nothing can reach Redis. Every service that tries to connect gets a timeout. The developer of the checkout service sees their connection to Redis failing and files a ticket. The debugging trail: checkout’s CiliumNetworkPolicy has no egress rule for Redis (because checkout declared redis: outbound but there’s no matching inbound on a MeshMember). The error: “outbound dependency redis declared but no matching inbound declaration found.”

The fix: the operator creates the LatticeMeshMember. The mesh-member controller matches bilateral agreements. Policies are generated. Traffic flows.

The lesson: the escape hatch fails closed, not open. In a poorly designed platform, an escape hatch might bypass security — a package deploys without policies and is reachable by everyone. In this architecture, default-deny is the fail-safe: a package without a MeshMember is unreachable, not unprotected. The escape hatch’s failure mode is “nothing works until you declare intent” — the same failure mode as a new LatticeService with missing dependency declarations. This is the security-first design paying off.

But it fails silently — the LatticePackage status says Ready (Redis is running) while no service can reach it. The platform should warn when a package has no corresponding MeshMember: “LatticePackage redis is Ready but no LatticeMeshMember exists in namespace data. Services cannot reach this package through the bilateral agreement model.”

11.8 Part III Complete

The derivation pipeline is fully described:

• Chapter 8: The pipeline — image verification, Cedar authorization, workload compilation, layered application, mesh readiness.
• Chapter 9: Secrets — five routing paths, backend abstraction, compile-time authorization, rotation.
• Chapter 10: Autoscaling, quotas, and cost — resource governance at derivation time.
• Chapter 11: Packages and escape hatches — LatticePackage for Helm, LatticeMeshMember for mesh integration, validation for policy.

Part IV shifts to security: the policies and enforcement mechanisms that the derivation pipeline evaluates and produces.

Exercises

11.1. [M10] A LatticeMeshMember declares allowedCallers: [checkout, inventory]. But checkout’s LatticeService spec doesn’t declare redis as an outbound dependency. Traffic from checkout to redis is denied by default-deny. Who needs to change — the LatticeMeshMember, the LatticeService, or both? What’s the error message the developer sees?

11.2. [H30] The LatticePackage supports values interpolation for secrets (${resources.redis.auth-token}). But Helm charts have their own secret management patterns — some charts create their own Secrets, some expect pre-existing Secrets, some use init containers to fetch from Vault. How does the platform’s secret routing interact with the chart’s own secret management? Design the boundary: what does the platform handle, what does the chart handle, and where do conflicts arise?

11.3. [R] Section 11.1 says “everything else uses LatticePackage + LatticeMeshMember.” An adversary argues: “This means any workload that doesn’t fit your three CRD types loses the full derivation pipeline — no automatic network policies, no compile-time authorization, no quota enforcement. The escape hatch is a second-class citizen.” Evaluate this argument. Is the gap acceptable? Could you close it further without building a new CRD for every workload type?

11.4. [H30] Platform infrastructure (Cilium, Istio, cert-manager) is installed during bootstrapping (Chapter 5) before the LatticePackage controller exists. Once the controller is running, should these components be managed as LatticePackages? What are the advantages (unified lifecycle management) and risks (a bug in the package controller could break the platform’s own infrastructure)?

11.5. [M10] A team wants to install a Helm chart that creates ClusterRoles and ClusterRoleBindings. The LatticePackage controller runs with namespace-scoped permissions. How should the platform handle charts that require cluster-scoped resources? Should the controller be granted cluster-admin? Should cluster-scoped charts be rejected? Design the authorization boundary.