AEM DevOps Performance sign-off in the Cloud Manager pipeline and after the release, against field data instead of scores
Cloud Manager checks deployments through the full-stack pipeline of AEM as a Cloud Service with code scanning, tests and a Lighthouse run. That is not a sign-off on load time: the checks run in the lab, on a few pages of the stage environment, and the Lighthouse score does not stop the pipeline. This page describes what the Cloud Manager quality gates do, where they end, and how budgets per page type, release markers in real user monitoring and rollback criteria defined in advance close the gap.
Cloud Manager pipeline: code scanning, functional tests and Experience Audit before production
A production pipeline in AEM as a Cloud Service runs in three phases: stage deployment, stage testing, production deployment. According to Adobe, only the full-stack pipeline supports code scanning, functional testing, UI testing and Experience Audit (Deploy Your Code). Before the production deployment, a user with the Business Owner, Project Manager or Deployment Manager role approves it, and production receives the same build artifact that was tested on stage.
| Gate | What is checked | Effect |
|---|---|---|
| Code Quality | rules based on SonarQube, such as security rating and test coverage | critical findings fail the pipeline, important ones pause it |
| Experience Audit | median Lighthouse scores for up to 25 configured paths, only the homepage without configuration | informational, does not stop the pipeline |
| Performance Testing | 30-minute load test on stage against KPIs for response time and page views per minute | part of Cloud Manager for Managed Services, critical findings fail the pipeline |
Adobe describes the thresholds for code quality and the load test under Code Quality Testing: a security rating below B is critical, test coverage below 50 percent is important. In the load test, an error rate of 2 percent or more is critical, and a 95th percentile response time above the KPI defined for the program is important. Details on the Experience Audit are in the Experience Audit Dashboard documentation.
Limits of the quality gates: Lighthouse score, 25 pages and lab instead of field
- Score instead of metric: The Experience Audit reports Lighthouse scores and their change from the previous run. A regression becomes visible but stops no deployment.
- Variability: Lighthouse results vary even without code changes. According to the Lighthouse documentation on variability, the median of five runs is about twice as stable as a single run.
- Few pages: Up to 25 paths do not cover every page type in every market on a large platform.
- Stage instead of production: The audit checks the stage environment, without consent and without interaction. Tags that load after consent and responsiveness to real input are missing, so Interaction to Next Paint cannot be checked this way.
- The load test measures the server: Response time and error rate under load say nothing about what happens in the browser afterwards.
A platform can therefore pass every check in Cloud Manager and still have poor real-user data. What Adobe's tool adds in operations is covered under AEM Sites Optimizer, the layers of the platform under Adobe Experience Manager Performance.
Performance budgets per page type as an additional gate in the pipeline
A budget defines per page type how much JavaScript, CSS and image data a page may load and which lab values for Largest Contentful Paint and Total Blocking Time serve as limits. Unlike a score, it can be assigned to a team and a component. How budgets are structured is described in Performance Budget.
In AEM as a Cloud Service, there are two ways to make a budget testable. Cloud Manager's custom UI tests run in a Docker image and allow Node and WebdriverIO besides Java; a measurement run against stage that fails when the budget is exceeded fits there. Alternatively, a separate check runs in version control before the merge. The raw Experience Audit data can be downloaded per page as JSON and is suitable for a time series across all releases.
The same check can query technical SEO attributes per page type: status code, canonical and the X-Robots-Tag header. Which ones matter in AEM is described in AEM SEO.
Sign-off after the release: field data per page type with release markers in RUM
The actual sign-off happens after deployment, with real visitors. For this, real user monitoring needs the version number of the deployment as an attribute of every measurement. Only then can the 75th percentile per page type be compared before and after a release, and a regression be attributed to a deployment.
The Chrome User Experience Report, which Google uses for its assessment, averages over 28 days and shows the full effect of a release only after about four weeks. It is unsuitable for a decision the day after a release, but it is the benchmark for the final sign-off. AEM's Operational Telemetry samples one page view in a hundred by default; on page types with little traffic, that is often not enough for a statement the next day.
Release markers also belong on changes that do not pass through Cloud Manager: new rules in Adobe Experience Platform Tags, activities in Adobe Target, a modified consent banner. Many regressions in Interaction to Next Paint come from these sources rather than from the code deployment.
Rollback criteria: thresholds per page type defined before the release
Whether a release is rolled back should be settled before the deployment, not debated afterwards. Three specifications per page type are enough:
- Threshold: how much the 75th percentile of a metric may rise compared with the state before the release, derived from the budget and Google's thresholds.
- Minimum data: how many measurements and which period are required before the threshold applies, so that noise does not trigger a rollback.
- Action: rollback by running the pipeline again with the previous state, disabling a tag rule, or a fix in the next release, with named responsibility.
How to phrase such criteria as testable tickets is shown in the Core Web Vitals acceptance criteria.
AEM on-premises and Managed Services: Jenkins or GitLab with Lighthouse CI
For self-operated AEM 6.5, there is no Cloud Manager; the pipeline runs in Jenkins, GitLab CI or a similar tool. Lighthouse CI can be added there as a separate step: several runs per URL via numberOfRuns, limits as assertions per audit, and aggregation by median instead of the best run. The URL list follows the page types from the budget, not the homepage alone. For the sign-off after the release, the same applies as in the cloud: release markers in RUM and field data per page type. For dates and paths beyond AEM 6.5, see AEM 6.5 end of support.
Acceptance criteria for the AEM pipeline: measurable per release
- A budget is defined for every page type and built into the pipeline as a check that fails when exceeded.
- The Experience Audit paths cover the most important page types, not just the homepage.
- Every measurement in real user monitoring carries the deployment's version number, including changes in Tags and Target.
- Rollback criteria per page type are documented before the release.
- Four weeks after the release, sign-off takes place against the Chrome User Experience Report.
Book a free performance analysis
30 minutes via video call. You see your platform's Google user data and which checks your pipeline lacks.
Frequently asked questions about AEM DevOps and performance
Does the Experience Audit in Cloud Manager stop a deployment with poor scores?
No. Adobe provides the Experience Audit as a quality gate that does not block deployments. It shows the median Lighthouse scores for the configured pages and the change from the previous run. To enforce a limit, add a separate check, for example in the UI tests or before the merge.
How many pages does the Experience Audit check?
Without configuration, the homepage of the publish instance; with configuration, up to 25 custom paths. At least one URL per important page type and market makes sense, selected by traffic and revenue.
Is a Lighthouse score from the pipeline enough for sign-off?
No. Lighthouse measures in the lab, without consent and without real input, and the score varies between runs. Google assesses field data over 28 days. Sign-off therefore takes place after the release against real user monitoring and the Chrome User Experience Report, per page type.
Who defines the rollback criteria?
Platform ownership together with the team responsible for implementation, before the release. Thresholds per page type, minimum data and the review after the release can be provided externally; the decision on a rollback and its implementation remain with the client team.