AEM Application Maintenance Tips and Tricks
In today’s world, Adobe Experience Manager (AEM) is a comprehensive content management solution for building websites, mobile apps and forms. And it makes it easy to manage your marketing content and assets. Adobe Experience Manager can be a real challenge to architect, deploy, and integrate.
A prime factor here is, you need to understand how your AEM application is performing under certain conditions. This is best done by monitoring the application over a long period of time.
When it comes to managing and maintaining an AEM application, it’s not just system maintenance that we should be concerned about, we should also think of it from the application code perspective.
AEM application maintenance is two faceted. One is ongoing, regular maintenance which falls under AEM DevOps/AEM Administrator tasks. Second is following the best practices in application development.
Here we are going to list out the various aspects of application maintenance from both system and development point of view:
DevOps/Admin Tasks
Online Revision Cleanup:
Every update to the repository such as creating, modifying, or publishing a page/asset, running workflows etc., creates a new content version which gradually increases repository size. In order to improve the AEM application performance and avoid repository growth, the old versions need to be cleaned up to free disk resources. This operation is called Online Revision Cleanup which have been there since AEM 6.3.
Unlike Offline Revision Cleanup (aka Offline Compaction), Online Revision Cleanup doesn’t require AEM instance to be shut down. Online Revision Cleanup is enabled by default to run every day. However, the execution time can be adjusted so that it runs in off-peak hours when there’s minimal activity on the system.
You can find Online Revision Cleanup under Tools > Operations > Maintenance > Daily Maintenance Window.
Note: There’s always a chance that running Online Revision Cleanup for the first time may not reclaim any space. The reason is, Online Revision Cleanup reclaims old revisions by generations. A fresh generation is generated every time Online Revision Cleanup runs. Only the content which is at least two generations old, will be reclaimed. That simply means, on the first run, there’s nothing to reclaim.
Offline Revision Cleanup:
Offline Revision Cleanup (aka Offline Compaction) also helps in reclaiming disk resources by cleaning up old revisions. However, in this case, AEM instance needs to be shut down to free up the space. It needs to be run manually using a set of commands and oak-run JAR.
So, what is the benefit of Offline Revision Cleanup? With Offline Cleanup, you can reclaim more space because the online mode keeps one generation while offline mode keeps two generations.
Note: Offline Revision Cleanup should be used only on an exceptional basis.
Workflow Purge:
Whenever we upload an asset or modify its metadata, it triggers a workflow. And this is not the only instance which triggers a workflow in AEM, there are many more cases. Along with OOTB workflows, we create various custom workflows during the development cycle for different purposes. Once these workflows are complete, they get archived and never get deleted from system. Hence, workflow purging is one of the mandatory tasks in AEM maintenance to increase the performance of the workflow engine.
Note: Workflow purging can be done for completed as well as running workflows.
You can find Workflow Purge under Tools > Operations > Maintenance > Weekly Maintenance Window.
Clicking on the Settings icon will take you to Workflow Purge configuration window in OSGi console. Workflow Purge is a factory configuration which allows you to configure multiple purge configurations for different models and workflow status.
Version Purge:
AEM creates a version of a page or an asset when you can activate the content after modifying it. You can also do this manually using the Timeline tab in sidebar.
These versions never get purged and can be restored at any point of time. The repository size grows slowly over the time because of these versions and therefore they need to be cleaned up to free up the disk resources.
These versions can be purged through Version Purge maintenance task. This task can be scheduled to remove the old versions automatically.
Audit Log Purge:
AEM events that qualify for audit logging generate more archived data. This data can quickly grow over time due to replications, asset uploads, asset deletion, page creation/modification, and other system activities.
This event data can be purged through OOTB Audit Log Purge maintenance task.
You can find Online Revision Cleanup under Tools > Operations > Maintenance > Weekly Maintenance Window.
Clicking on the Settings icon will take you to Audit Log Purge configuration window in OSGi console. There are three types of log purge options:
DAM Audit log Purge Rule:
Pages Audit log Purge Rule:
Replication Audit log Purge Rule:
Generating Heap dump:
There are cases when your AEM application may go down all of a sudden after performing slow over a period of time and then runs out of memory. Such problems occur due to various reasons; one possible reason is, JAVA process was started with default heap memory settings. That means the JVM parameter -Xmx was not specified.
If that is not the case, then your AEM application might be retaining so many objects and not releasing them for Garbage Collection. This will cause a memory leak. This issue can be identified through Heap Dumps.
You just need to make sure that your system is configured to generate heap dumps. Heap dump can help you analyze application issues such as:
- Out of Memory Error
- Frequent garbage collections
You can either configure AEM to automatically generate heap dump when “Out of Memory” error occurs, or you can take it manually through OSGi console.
In order to automate it, you would need to add below JMX parameters to your startup script:
- -XX:+HeapDumpOnOutOfMemoryError
- -XX:HeapDumpPath=/path/for/generating/heapdump
On the other side, to generate heap dump manually, go to “/system/console/memoryusage”. Here you can either generate and download heap dump instantly by clicking “Dump Heap” or you can generate it at a regular interval.
Generating Thread dump:
Thread Dump is a list of JAVA threads that are active in a JVM at a certain period of time. When you want to analyze your application, it is recommended to take 10 thread dumps at regular intervals for e.g. 1 thread dump every 10 seconds.
You can generate a thread dump by using below command:
jstack -l <java_pid> >> threaddump.log
Once you have the thread dumps, you can use any third-party tool to analyze it. There are several third-party tools available.
Few other items to help you improve the AEM application performance:
- Enabling Minification and GZip
- In order to improve the page load time, you should minify JS and CSS files and Gzip them before delivering. Minification actually minifies JS and CSS files using YUI compressor.
- Check the dispatcher configuration
- Check the “/cache” section to see what all things are being cached and what all things get invalidated when you publish a page or an asset.
- Check statfile configuration on dispatcher.
- You need to make sure that your dispatcher is configured in such a way that it caches most of the documents and it passes very few requests to the Publish instance.
Development Tasks
Review all OOTB DAM workflows:
Workflows enable us to automate most of the activities in AEM. It’s an important processing part in AEM and needs to be configured according to the best practices, if not, there can be a major impact on the system performance. Hence, it is highly recommended to plan your workflows implementations carefully.
Minimize the number of launchers in AEM. There are listeners that are responsible for all of the registered workflow launchers: It will listen to all the changes on all of the paths specified in the globbing properties of the other launchers. When an event is dispatched, the workflow engine evaluates every launcher to determine if it should run.
Creating too many launchers will cause the evaluation process to run slowly. For this reason, it is recommended to create launchers only when needed and make the globbing path as specific as possible.
Note: Please disable any out-of-the-box launchers that are not in use.
Do not start Workflows from other Workflows. Workflows can carry a significant amount of overhead, both in terms of objects created in memory and nodes tracked in the repository. For this reason, it is better to have a workflow do its processing within itself rather than starting additional workflows.
Transient workflows:
When a workflow is transient, the runtime data related to the intermediate work steps is not persisted in the JCR whenever it is in processing. To optimize high ingestion loads, we can define a workflow as transient.
Wherever possible, set the DAM Update Asset workflow to transient. The setting significantly reduces the overheads required to process workflows because, in this case, workflows need not pass through the normal tracking and archival processes.
The advantages of transient workflows can include:
- A reduction in the workflow processing time; up to 10%.
- Significantly reduce repository growth.
- Reduces the number of TAR files to compact.
How to make the DAM Update Asset workflow Transient in AEM:
- Open http://localhost:4502/miscadmin on the AEM instance you want to configure.
- From the navigation tree, expand Tools > Workflow > Models > dam.
- Double-click DAM Update Asset.
- From the floating tool panel, switch to the Page tab, and then click Page Properties. Select Transient Workflow Click OK.
Query Performance Analysis:
Queries lacking a nodetype restriction force AEM to assume the nt:base nodetype, which every node in AEM is a subtype of, effectively resulting in no nodetype restriction.
Setting type=cq:Page or type=dam:Asset restricts this query to only cq:Page nodes or dam:Asset nodes, and resolves the query to AEM’s cqPageLucene, damLucene, limiting the results to a subset of nodes in AEM.
For cases where query execution is fast but the number of results are large, p.guessTotal is a critical optimization for Query Builder queries. p.guessTotal=100 tells Query Builder to collect only the first 100 results, and set a boolean flag indicating if at least one more results exist.
Oak Index Management:
Due to AEM’s flexible content architecture, it is difficult to predict and ensure that the traversals of content structures will not evolve over time to be unacceptably large. Therefore, ensure an index satisfy queries, except if the combination of path restriction and nodetype restriction guarantees that less than 20 nodes are ever traversed. One index is the Property Index, for which the index definition is stored in the repository itself. Implementations for Apache Lucene and Solr are also available by default, which both support full text indexing. The Traversal Index is used if no other indexer is available. This means that the content is not indexed, and content nodes are traversed to find matches to the query. If multiple indexers are available for a query, each available indexer estimates the cost of executing the query. Oak then chooses the indexer with the lowest estimated cost.
AEM’s internal re-indexing process collects repository data and stores it in Oak indexes to support performant querying of content. In exceptional circumstances, the process can sometimes become slow or even stuck. It is important to distinguish between re-indexing that takes an inappropriately long amount of time, and re-indexing that takes a long amount of time because of its indexing vast quantities of content. For example, the time it takes to index content scales with the amount of content, so large production repositories will take longer to re-index than small development repositories.
Why Outsource NextRow Digital for AEM Application Maintenance?
AEM has never been a one-size-fits-all system, and today more-so than ever. Here at Nextrow its fabulous that we do now have such a wealth of options on how to host AEM, and it’s important for anyone who has a stake in the game to weigh their options carefully as to where they’re going to put what will end up being their very most valuable pages and marketing assets.
If you’d like to know more about our AEM Managed Services, you can reach our sales team at +1-847-592-2920 or email us to sales@nextrow.com. The AEM experts at NextRow Digital would be happy to answer all your queries.
