k8s-objectmatcher Administrator & Developer Guide

This guide is the deep, source-based reference for cisco-open/k8s-objectmatcher. The library is small — one Go package (patch), four production source files, two test files, ~25 KB of source — but it sits in the critical path of Kubernetes controllers. A bug in the diff calculation can silently turn a healthy reconciler into a hot loop that thrashes the API server, or worse, into a quiet reconciler that fails to apply real changes. This document covers the architecture in detail, the full configuration surface, the patch-calculation algorithm step by step, the integration risk model, and the developer workflow for adding support for new resource kinds or custom cleaners.

The audience is operator authors, controller-runtime users, library maintainers, and security reviewers tracing how patch payloads flow through the system.

1. Architecture Overview

k8s-objectmatcher is a pure-function diff engine with two side effects on Kubernetes objects: stamping a last-applied annotation and reading it back. Everything else — the strategic-merge / JSON-merge dispatch, the per-kind cleaners, the null-stripping pass — operates on in-memory []byte buffers without touching the cluster.

1.1 What the library does

1. Marshals both the current (live) object and the desired (modified) object into JSON.
2. Applies a configurable chain of CalculateOption filters that delete or normalise fields on both byte streams symmetrically.
3. Strips nulls and zero-valued IntOrString types via DeleteNullInJson.
4. Reads the last-applied snapshot from the live object's annotation (banzaicloud.com/last-applied by default), which the controller itself stamped on the previous reconcile.
5. Computes a three-way merge patch — strategic for typed objects, JSON-merge for *unstructured.Unstructured.
6. Re-applies the patch locally and re-diffs to suppress $setElementOrder noise on the strategic-merge path (and the JSON- merge path runs the same trick).
7. Returns a *PatchResult with the patch, the cleaned current bytes, the cleaned modified bytes, and the original snapshot.

The controller compares PatchResult.IsEmpty() against "{}" and either skips the update (in-sync) or calls client.Update (real change).

1.2 ASCII architecture diagram

                                                                    
                                                                    
                                  Controller Reconcile Loop          
                                                                    
                                                                    
                                                                    
                                                                    
        currentObject (from API)                                     
        modifiedObject (built locally)                               
                                                                    
                                                                    
                                                                    
                              patch.PatchMaker                       
                                                                    
                                                                    
                                                                    
                                                                    
        1.  json.Marshal(currentObject)                              
        2.  json.Marshal(modifiedObject)                             
        3.  apply CalculateOption chain                              
            IgnoreStatusFields                                       
            IgnoreField                                              
            IgnoreVolumeClaimTemplateTypeMetaAndStatus               
            (custom)                                                 
        4.  DeleteNullInJson on both                                 
        5.  Annotator.GetOriginalConfiguration(currentObject)        
                                                                    
                                                                    
                                                                    
                                                                    
                          type switch on currentObject               
                                                                    
                                                                    
                                                                    
                                                                    
        unstructured.Unstructured              everything else       
                                                                    
                                                                    
                                                                    
        BaseJSONMergePatcher           K8sStrategicMergePatcher      
        CreateThreeWay                 CreateThreeWayMergePatch      
        JSONMergePatch                 then apply locally and        
        then apply locally,            CreateTwoWayMergePatch        
        CreateMergePatch               to collapse                   
                                       setElementOrder noise         
                                                                    
                                                                    
                                                                    
                                                                    
                              PatchResult                            
                              Patch                                  
                              Current                                
                              Modified                               
                              Original                               
                                                                    
                                                                    
                                                                    
                                                                    
        IsEmpty()                                                    
                                                                    
                                                                    
                                                                    
                                                                    
        true                              false                      
        skip update                       client.Update              
                                          SetLastAppliedAnnotation   
                                                                    
                                                                    

1.3 Component inventory

Note

The init() block in patch/deletenull.go overrides the global json-iterator encoder for intstr.IntOrString. Importing the patch subpackage anywhere in your binary changes the marshal behaviour of json-iterator everywhere — this is intentional (so Calculate sees stable bytes) but worth knowing if you also use json-iterator elsewhere.

1.4 Technology stack

The integration test module (tests/) is a separate Go module with its own go.mod and pulls in k8s.io/client-go v0.21.3, k8s.io/api v0.21.3, and k8s.io/apiextensions-apiserver v0.21.3 — these are not runtime dependencies of patch.

1.5 Data flow for the main workflow

  Controller                Annotator             PatchMaker            Patcher impls
                                                                
  Reconcile(req)                                                
    Get(current) <-- API --                                     
    desired = build()                                           
                                                                
    SetLastApplied(desired) -> GetModified(desired,false)       
                            <- modifiedNoAnn                    
                            -> SetOriginal(desired,modifiedNoAnn)
                                                                
    Calculate(current, desired, opts...)                        
      json.Marshal(current)                                     
      json.Marshal(desired)                                     
      apply each opt                                            
      DeleteNullInJson * 2                                      
      GetOriginal(current) <-- read annotation, base64+unzip ---
                                                                
      if *unstructured.Unstructured                             
         CreateThreeWayJSONMergePatch    -> jsonMergePatcher    
         if patch != "{}":                                      
           MergePatch                    -> jsonMergePatcher    
           CreateMergePatch              -> jsonMergePatcher    
      else                                                      
         CreateThreeWayMergePatch        -> strategicMergePatcher
         if patch != "{}":                                      
           StrategicMergePatch           -> strategicMergePatcher
           CreateTwoWayMergePatch        -> strategicMergePatcher
                                                                
    return PatchResult{Patch, Current, Modified, Original}      
                                                                
    if IsEmpty(): return                                        
    Update(desired) -- API ->                                   

2. System Requirements

Because the library is a Go module with no daemon and no CLI, the only runtime requirements are those of the controller binary that imports it.

2.1 Build requirements

2.2 Runtime requirements (from the calling controller)

Note

The library's pinned apimachinery v0.19.16 in go.mod is the baseline for strategic-merge metadata. When a controller imports apimachinery v0.30+, Go's MVS picks the newer version and that's what's actually compiled. The pin only matters for builds that downgrade — which is rare.

2.3 Object-size constraints

The annotation system imposes Kubernetes' standard 256 KiB per-annotation cap. The library compresses with archive/zip before base64 encoding, so the effective ceiling is roughly 1–2 MiB of raw JSON (compression ratio depends on object content; ConfigMaps with text-heavy data compress well, dense binary fields don't).

3. Configuration Reference

The library has no config file and no environment variables. Configuration happens at the call site — option arguments to Calculate and constructor arguments to NewAnnotator / NewPatchMaker. This section documents every knob.

3.1 CalculateOption reference

Application order matters because each option sees the byte stream mutated by the previous one. Order is the slice index in the opts ...CalculateOption variadic argument to Calculate.

Important

IgnoreField only affects top-level keys. To suppress a nested field (e.g. spec.replicas, metadata.labels[someKey]), you must implement a custom CalculateOption. This is a known gap — see #57.

3.2 Annotator configuration

NewAnnotator(key string) is the only constructor. The single config input is the annotation key. Defaults:

Important

Changing the annotation key in a running operator orphans the existing snapshots on every live object. Plan a backfill: read every managed object, call SetLastAppliedAnnotation with the new key, and Update. Until the backfill completes, those objects' diffs degrade to two-way merges and may produce large, server-defaulted patches.

3.3 PatchMaker configuration

NewPatchMaker(annotator *Annotator, smp StrategicMergePatcher, jmp JSONMergePatcher) Maker takes three injected dependencies:

The interfaces are deliberately small so you can substitute fakes in tests or wrap the defaults with metrics / tracing.

3.4 K8sStrategicMergePatcher.PreconditionFuncs

The default strategic-merge implementation has a single configurable field:

type K8sStrategicMergePatcher struct {
	PreconditionFuncs []mergepatch.PreconditionFunc
}

Precondition functions run on the patch and can short-circuit the merge with an error. The two stock functions in k8s.io/apimachinery/pkg/util/mergepatch are:

• RequireKeyUnchanged(key) — error if the patch attempts to change the named key.
• RequireMetadataKeyUnchanged(key) — same, but scoped to metadata.

maker := patch.NewPatchMaker(
	patch.DefaultAnnotator,
	&patch.K8sStrategicMergePatcher{
		PreconditionFuncs: []mergepatch.PreconditionFunc{
			mergepatch.RequireKeyUnchanged("apiVersion"),
			mergepatch.RequireMetadataKeyUnchanged("uid"),
		},
	},
	&patch.BaseJSONMergePatcher{},
)

These run only on the CreateThreeWayMergePatch and CreateTwoWayMergePatch paths. Use them to harden against logic bugs that would otherwise produce a patch that mutates an immutable field.

3.5 The full configuration surface, in one table

There are no other knobs. No timeouts (the library doesn't make network calls), no logger injection (errors are returned, not logged), no rate limits.

4. Deployment & Scaling

Deployment of k8s-objectmatcher is deployment of the controller binary that imports it. There is nothing to deploy separately.

4.1 Build deployment checklist

• [x] go get github.com/cisco-open/k8s-objectmatcher/patch
• [x] Pin the version in go.mod (don't track latest in production controllers)
• [x] Run go mod vendor if your build is hermetic
• [x] Cross-compile to your target arch (Linux/AMD64 and Linux/ARM64 both work without changes)
• [x] Build the controller container image as usual
• [x] Deploy via your normal operator-distribution channel (Helm, OLM, raw manifests)

4.2 Scaling considerations

The library is stateless and pure — there is nothing to scale horizontally or vertically. Performance characteristics:

For controllers reconciling thousands of objects per second, the GC allocations are usually the limiting factor. There is no built-in pooling of []byte or map[string]interface{} buffers; consider profiling with -memprofile and a sync.Pool wrapper if you hit hot-path GC pressure.

Tip

If you reconcile the same object frequently and the snapshot rarely changes, cache the PatchResult.Patch == "{}" decision at the controller layer keyed by (uid, resourceVersion). The library itself does no caching — by design, since determining whether resourceVersion changes equate to semantic changes is the very question the library exists to answer.

4.3 Versioning policy

The repo follows Go module versioning. Major version bumps would require a path change to /v2 per the Go module rules. As of this writing, the library lives at the v0/v1 module path (github.com/cisco-open/k8s-objectmatcher).

5. Operations

Operating a controller that uses this library is operating any other controller — there's no dedicated runbook for the library itself. That said, three operational concerns deserve their own subsections.

5.1 Observability

The library does not log. Every error is returned wrapped via emperror.dev/errors and propagated up to the caller. The controller is responsible for logging:

result, err := maker.Calculate(cur, des, patch.IgnoreStatusFields())
if err != nil {
	logger.Error(err, "patch calculation failed",
		"kind", cur.GetObjectKind().GroupVersionKind().Kind,
		"name", cur.GetName())
	return err
}
if !result.IsEmpty() {
	logger.V(1).Info("applying patch",
		"patch", string(result.Patch),
		"size", len(result.Patch))
}

Note

Logging result.Patch at V(1) is invaluable for debugging — but see Section 7.3 below for the security implication: patches can contain secret material if your controller manages Secrets or Secret-bearing objects. Log them at V(2) and through a redacting log sink in production.

5.2 Annotation hygiene

The last-applied annotation grows with the object. For a Deployment or StatefulSet, expect 2–10 KiB of base64-encoded zip per object. Per-namespace etcd consumption can become noticeable at thousands of objects, but the per-object annotation cap (256 KiB) is the more important limit.

5.3 Upgrading the library in a running operator

Drop-in safe for the supported version line. The annotation format has been stable since the zip+base64 migration, and the read path keeps a fallback for plain JSON values, so upgrading from any older Banzai Cloud or cisco-open build works without flag days.

The strategic-merge / JSON-merge dispatch table is also stable. The public surface — PatchMaker, Annotator, CalculateOption, PatchResult — is treated as part of the contract.

Tip

When upgrading, run a go test ./patch/... against a unit-test harness with representative objects from your production cluster. The library has 100% public-API coverage in its own tests; layered tests in your controller catch any subtle behaviour changes that matter to your CRDs.

6. Performance

The hot path is Calculate. Cost analysis:

6.1 Per-call cost breakdown

A typical Deployment runs Calculate in 1–3 ms on modern hardware. StatefulSets with many volumeClaimTemplates are slower because IgnoreVolumeClaimTemplateTypeMetaAndStatus walks the templates list in addition to the rest of the spec.

6.2 Allocation profile

The dominant allocations are:

1. The map[string]interface{} returned by json.Unmarshal inside each option and inside DeleteNullInJson.
2. The []byte slices returned by every Marshal.

A single Calculate against a Deployment allocates ~30–50 KiB of heap. The GC handles this fine in normal operation; controllers running at >1000 reconciles/sec/CPU may want to look at hot-path profiling.

Note

The library does not pool buffers internally. If you need pooling, wrap the Maker interface in your own implementation that pools the []byte outputs of Calculate for downstream logging.

6.3 Benchmarking

There are no shipped benchmarks. To measure your own:

func BenchmarkCalculate(b *testing.B) {
	cur := buildDeployment("cur")
	des := buildDeployment("des")
	maker := patch.DefaultPatchMaker
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		_, _ = maker.Calculate(cur, des, patch.IgnoreStatusFields())
	}
}

Run with go test -bench=. -benchmem ./yourpkg/....

7. Security

k8s-objectmatcher has no network surface — it never opens a socket, never makes an HTTP call, never reads a file. The security posture is therefore framed in terms of integration risk (what happens when the calling controller misuses the library) and data exposure (what's in the patch payloads that leave the library's address space).

7.1 Integration risk: the buggy-patcher problem

A bug in patch calculation propagates to a destructive reconcile. Two failure modes are worth calling out:

The first failure is loud and self-correcting (you'll notice in metrics and back off). The second is silent and dangerous. Custom CalculateOptions must be symmetric — mutate current and modified the same way — and unit-tested with deliberately mutated inputs.

Caution

A custom CalculateOption that deletes a field only on the modified side will silently mask real updates. Always assert in tests that the option produces identical mutations on both byte slices for inputs that are equal.

7.2 Operator security posture

The library inherits the operator's RBAC and the operator's threat model. Specifically:

• The operator typically holds verbs get, list, watch, create, update, patch, delete on the resources it manages.
• A bug in the library (or in the controller's use of it) cannot exceed those verbs — Calculate produces bytes, but only the controller's client.Update call commits them.
• The annotation it stamps is a benign metadata field. It does not carry capability or trust beyond what's already in the object.

Warning

Do not treat the last-applied annotation as a source of truth for security decisions (e.g. "this object was previously approved because it has our annotation"). Annotations can be written by any principal with update on the object — including humans with kubectl access. The annotation is a controller-internal cache, not an authentication signal.

7.3 Patch payloads in audit logs and observability pipelines

The patch bytes that flow through Calculate are the diff of two Kubernetes objects, which means they can carry sensitive material for any object type that holds secrets:

If your controller logs result.Patch (a useful debugging practice) and your log pipeline forwards everything to a centralized store (Loki, Splunk, Datadog), you have just exfiltrated secret material to that pipeline. Same for the audit log if you kubectl describe the patch payload through some intermediary.

Recommendations:

1. Log result.Patch only at high verbosity (e.g. V(2) in logr/klog).
2. Configure a redacting log sink that strips Secret.data, ConfigMap.data for known-sensitive keys, and env values matching *_PASSWORD, *_TOKEN, *_KEY patterns.
3. For controllers that manage Secret objects, consider replacing the patch payload with a hash for log purposes:

import "crypto/sha256"

if !result.IsEmpty() {
	hash := sha256.Sum256(result.Patch)
	logger.V(1).Info("patch needed", "patch_sha256", fmt.Sprintf("%x", hash[:8]))
}

Caution

Audit logs and tracing spans are also exposure surfaces. result.String() formats the entire Patch, Current, Modified, and Original — that's four copies of the (possibly secret- bearing) object, one of them base64-decoded from the annotation. Never put result.String() in a span attribute.

7.4 Supply chain

The library has a small, well-known dependency graph:

The license inventory is enforced by .licensei.toml (Apache-2.0, MIT, BSD, MPL-2.0). License audits are routine — see the file for the exact ignore list.

8. Developer Internals

This section documents the internals so contributors can extend the library without breaking the contract.

8.1 Repository structure

8.2 The strategic-merge flow, step by step

The flow inside (*PatchMaker).Calculate for a typed object (the "everything else" branch of the type switch):

1. Marshal current and modified with json-iterator (which uses the custom IntOrString encoder registered in init() — so IntOrString{IntVal: 0, StrVal: ""} becomes null, not 0).
2. Apply each CalculateOption in order. Each option round- trips through Unmarshal + delete + Marshal, which means the per-option cost is O(N) regardless of how small the deletion is.
3. DeleteNullInJson walks the entire JSON tree and removes any key whose value is nil or zero-valued (per isZero — which treats float, int64, bool as never zero so they aren't accidentally elided when the user intentionally set 0/false).
4. Annotator.GetOriginalConfiguration(currentObject) reads the annotation. The implementation:
   • Reads annots[a.key].
   • Tries base64 decode. If the result has content type application/zip, unzips and returns the inner bytes.
   • If base64 decode fails, falls back to returning the raw annotation bytes (legacy plain-JSON path).
5. Type switch on currentObject:
   • *unstructured.Unstructured: route to (*PatchMaker).unstructuredJsonMergePatch.
   • default: continue with strategic merge.
6. strategicMergePatcher.CreateThreeWayMergePatch(original, modified, current, currentObject) produces the candidate patch.
7. If string(patch) != "{}":
   • Apply the patch to current locally: patchCurrent, _ := strategicMergePatcher.StrategicMergePatch(current, patch, currentObject).
   • Re-diff: patch, _ := strategicMergePatcher.CreateTwoWayMergePatch(current, patchCurrent, currentObject). This collapses $setElementOrder markers that strategic merge generates even for unchanged lists.
8. Build and return the PatchResult.

The unstructured path mirrors the same trick: CreateThreeWayJSONMergePatch → if non-empty, MergePatch to apply locally, then CreateMergePatch to re-derive a clean diff.

8.3 The per-kind cleaner pattern

A "cleaner" is a CalculateOption that knows about a specific Kubernetes kind and normalises a field that the API server mutates. The shipped cleaner is IgnoreVolumeClaimTemplateTypeMetaAndStatus, which targets StatefulSet spec.volumeClaimTemplates. The pattern:

func IgnoreXYZ() CalculateOption {
	return func(current, modified []byte) ([]byte, []byte, error) {
		current, err := mutateXYZ(current)
		if err != nil {
			return nil, nil, errors.Wrap(err, "...")
		}
		modified, err = mutateXYZ(modified)
		if err != nil {
			return nil, nil, errors.Wrap(err, "...")
		}
		return current, modified, nil
	}
}

func mutateXYZ(obj []byte) ([]byte, error) {
	var resource map[string]interface{}
	if err := json.Unmarshal(obj, &resource); err != nil {
		return nil, errors.Wrap(err, "could not unmarshal byte sequence")
	}
	// ...navigate into the structure, normalise the field...
	out, err := json.ConfigCompatibleWithStandardLibrary.Marshal(resource)
	if err != nil {
		return nil, errors.Wrap(err, "could not marshal byte sequence")
	}
	return out, nil
}

mutateXYZ is responsible for being defensive — every layer of type assertion (if spec, ok := resource["spec"].(map[string]interface{}); ok) must guard against the field not existing. Returning an unmodified byte stream when the field isn't present is the correct behaviour.

8.4 Adding support for a new resource kind

The high-level recipe to add an IgnoreFooBarBaz() option for a new kind:

1. Identify the server-mutated fields by kubectl get -o json the live object after creation and diffing it against the desired YAML you submitted. Note every key that appears in the live but not the desired.
2. Decide the mutation strategy — delete the field on both sides (use delete()), or normalise it to a fixed value (set it on both sides, like IgnoreVolumeClaimTemplateTypeMetaAndStatus does).
3. Implement following the cleaner pattern above.
4. Add a unit test in patch/deletenull_test.go (create the file if it doesn't exist) that:
   • Creates two byte streams with the field set.
   • Runs the option.
   • Asserts both streams come out identical and the field is gone (or normalised).
   • Asserts that for inputs without the field, the option is a no-op.
5. Test against Calculate end-to-end with a representative *unstructured.Unstructured and a typed object.
6. Document in this guide and the User Guide's option table.

Note

The tests/ directory at the repo root is a separate Go module with k8s.io/client-go and apiextensions-apiserver as dependencies. End-to-end tests for new options usually live there rather than in the patch/ unit tests, because they require a real (or envtest-simulated) API server.

8.5 Why the second pass exists

Strategic merge generates $setElementOrder/<list-name> directives whenever a mergeable list is involved, even when the contents and order are identical. That's because strategic merge's three-way algorithm computes the order from the merge of three versions and emits a directive when the logical order could differ — even if in practice it doesn't.

The library's response is to apply the patch back onto current and re-diff with two-way merge. If applying the patch yields the same current, the two-way diff is empty, and the patch collapses to {}. This is the key innovation that makes the library practical for real controllers; without it, every reconcile of any object with a list (Pods always have at least containers, volumes, etc.) would see a non-empty patch.

The unstructured path uses the same trick but with MergePatch (apply) and CreateMergePatch (re-derive). JSON merge doesn't have $setElementOrder semantics — but it does have its own quirks around explicit-null values, which the second-pass round-trip also smooths over.

8.6 The intstr.IntOrString json-iterator override

The init() block in patch/deletenull.go is the single largest behavioural override the library imposes on its callers' process. It registers a custom encoder for intstr.IntOrString:

• If IntVal == 0 and StrVal != "" && StrVal != "0" → write the string.
• If IntVal == 0 and StrVal == "" or StrVal == "0" → write null.
• Otherwise → write the int.

The companion IsEmpty predicate returns true only for the second case — so json-iterator omits the field entirely when serialising a struct with omitempty.

The motivation: Kubernetes uses intstr.IntOrString for targetPort, maxSurge, maxUnavailable, port, etc. Default json-iterator (and stdlib encoding/json) marshal an unset IntOrString as 0, which produces a false diff against a server representation that often serialises it as null or omits it. Forcing both sides to write null for the unset case eliminates that whole category of spurious diff.

Important

Importing github.com/cisco-open/k8s-objectmatcher/patch registers this encoder process-wide. Any other code in the binary that uses json-iterator to marshal an IntOrString will see the same behaviour. This is almost always what you want for Kubernetes code, but be aware of it.

8.7 Testing strategy

Run the production unit tests:

go test ./patch/...

ok  	github.com/cisco-open/k8s-objectmatcher/patch	0.123s

Run the integration suite (requires KUBECONFIG):

cd tests && go test ./...

8.8 Code conventions

From .golangci.yml:

• gocyclo min-complexity: 15
• linters enabled: misspell, gofmt, goimports
• linters explicitly disabled: golint, errcheck, gas, megacheck, varcheck, gochecknoglobals, govet (the last due to a known false positive about non-nil ≠ nil)

Errors are wrapped with emperror.dev/errors (not stdlib fmt.Errorf with %w) to attach contextual key/value pairs via errors.WrapWithDetails.

Every Go file carries the Apache-2.0 license header, with Copyright © :YEAR: Banzai Cloud. The header template is enforced by .licensei.toml.

8.9 Potential improvements

8.10 Contributing

Standard fork-and-PR flow against github.com/cisco-open/k8s-objectmatcher. The repo enforces:

1. Apache-2.0 license header on every Go file (template in .licensei.toml).
2. gofmt + goimports clean.
3. go test ./patch/... passes.
4. The integration suite in tests/ passes if your change touches anything that an API server interaction would observe.

Open issues are catalogued at github.com/cisco-open/k8s-objectmatcher/issues.

9. Integration Points

The library's only integration with the outside world is the calling controller. This section enumerates the integration surfaces and how each maps to a public API symbol.

9.1 Inbound integration: what callers depend on

9.2 Outbound integration: what the library depends on

9.3 Common integration patterns

9.4 Compatibility matrix

9.5 What the library does not integrate with

To set expectations and prevent looking-for-features-that-aren't-there:

Note

If you need interoperability with kubectl apply's last-applied-configuration annotation, you can swap the key with NewAnnotator("kubectl.kubernetes.io/last-applied-configuration"). The format kubectl writes is plain JSON (not zip+base64), and the Annotator read path falls back to plain JSON when base64 decode fails — so reading interoperates. Writing through this library will switch the annotation to zip+base64 on the next reconcile, which kubectl apply --prune does not understand. Mixing the two is generally inadvisable.