This Blog explains some attributes that Midships tunes across the ForgeRock Platform to meet the non-functional requirements that our Financial Services Customers have given us over the years. ForgeRock has proven time and time again to be a high-performing service and when tuned correctly can support in excess of 3,000 TPS (that's 259 million transactions a day).
I am Ravivarma and one of Midships' ForgeRock Senior SMEs. If you have any queries on this Blog, please feel free to get in touch with me at ravivarma.baru@midships.io
Caveats: Performance tuning depends on the deployment configuration including whether you are running services on VMs or Containers. The tuning suggestions mentioned below are generic and for consideration.
ForgeRock Directory Services
ServerId (server name)
Keep the server name small and identifiable (ex: USDC1). This is because the server name is used in every replication activity, for storing the sync state. It does have a material size on the size of logfiles and the DB.
Change Number Indexer disabling:
If no other 3rd party application (outside of AM, so yes IDM is a third party application) is querying your Directory Service e.g. User Store or CTS then you can disable the change number indexer. Refer to changelog-enabled:disabled.
This will increase the performance of Directory Service replication.
Replication Groups:
Pair the DS and RS in groups based on criteria like if they are in the same data centre for a faster replication changes (and reduce cross DC traffic).
DB Verifier Thread:
If the data size is vast, then change the schedule of the DB verifier thread that verifies the integrity of each backend in a DS to spread over different times in off-peak hours in different DSes. By default, it is scheduled at 12 AM on all the DSes.
FSyncs:
If the FSync delays are high (which can be observed in support extracts), better to upgrade the disks to better I/O disks.
Indexes:
Remove unwanted indexes. Many clients have indexes that include both equality and presence, but in most cases, the presence is never used. From DS 7.3.0, the startup shows a warning if both indexes are created and presence is not required. Those indexes can be removed which results in lesser storage of the data.
ForgeRock Access Manager
Session cache and JVM Settings
Set the maximum number of sessions that needs to be cached in AM, (used to enable AM to search through the session validity and properties in memory instead of going to the CTS) .
Configure > Server Defaults > Session > Session Limits, default value is 5000.
This can be tuned as per client production load (can be safely set to 100,000s).
Using G1GC, try to limit the GC pause time for minor GCs to 200ms, -XX:MaxGCPauseMillis=200
Increasing the session cache will require AM to have sufficient memory. This can be achieved by increasing the heap size, also heap size depends on other cache settings described below.
Ldap Connections:
For CTS: default max connections 10, tune it to 65 / 129 etc (2^n + 1 [one is reserved for reaper thread]).
For US: default max connection is 10, which can be increased to 64 or other value depending on the load and the resources available on the server.
For CFG: the default max connection is 10, which can be set to 64 or other value depending on the load and the resources available on the server.
For Nodes / Authentication Modules: default is set as 10, this can be tuned to 65 ForgeRock AM 7 > Maintenance Guide > Tuning LDAP Connectivity
Log level
Log-level must be set in ERROR level for AM deployments on production, otherwise we will see performance implications with Message or Info Level.
SDK caching
The maximum number of user entries cached can be set using SDK caching: Configure > Server Defaults > SDK. Default 10000, tune this value based on the load.
DN Caching
Setup DN caching, so that the user DNs are cached (applicable only do this if the user dns are not changed very frequently).
Configure > Server Defaults > SDK > caching and replica. Set the number of DNs to cache default is 1500.
Session Blacklist Cache Size
If using client-based sessions, with session blacklisting, then tune the session blacklist cache size to meet the performance requirements.
Configure > Global Services > Session > Client-Based Sessions > Session Blacklist Cache Size. This can be set to 100,000s based on the performance requirements and tests.
OAuth2 Provider Token Blacklist cache size
OAuth2 tokens blacklist can be stored in AM cache, to reduce the number of calls to CTS.
Configure > Global Services > OAuth2 Provider > Global Attributes > Token Blacklist Cache Size. This can be set to 100,000s based on the performance requirements and tests.
ForgeRock Identity Manager
Additional JVM Tuning
As IDM handles data that has common keys example DS data or DB data, using string deduplication can also help : -XX:+UseStringDeduplication
Sync Use Cases :
service accounts :
Ensure that correct privileges are given to service accounts that query or write data to the DS.
bypass-acl setting can be turned on the DS, which will remove the additional check for service account to increase throughput.
change numbers:
Allow IDM to read change numbers for live sync, which will have improved performance. Note does not work if you use TimeStamp-based LiveSync.
Indexes:
Index the fields with which IDM reads data from DS or DB frequently.
If livesync is used with timestamps, instead of change logs, index modify and createTimestamp in DS, so that IDM performance is optimal.
If IDM uses managed objects and uses DB as backend, use searchable as true (default true) for the required fields of Managed Object, which helps to create index in DB for the field.
Sync per objectclass:
IDM sync uses single thread and it cannot handle more requests per second. To make IDM handle multiple requests in parallel, configure sync per target objectclass, instead of using a single top level object class.
ForgeRock Identity Gateway
Resource release issue:
IG Custom Filters written in Java implements filter method, which returns a response with an object of class Promise. Promise is implemented asynchronously in ForgeRock Java APIs. We recommend you separate requests (that use the same filter and the default ForgeRock handlers) into different handlers & routes.
If a single filter makes two or more http calls using the ForgeRock Handler object asynchronously, then the resources will not be released by the ForgeRock OOB code. Example: check below filter method of a filter class:
public class ExampleFilter implements Filter {
public ExampleFilter(Handler handler, ....) {
}
public Promise<Response, NeverThrowsException> filter(Context context, Request, request, Handler handler) {
Headers headers = request.getHeaders();
//any handler, even if another handler like amHanlder used to the right, will still result in issue
Promise<Response, NeverThrowsException> promiseToBeReturned = handler.handle(
context, request).thenAlways(request::close).
thenAsync(new AsyncFunction<Response, NeverThrowsException>(){
Request newRequest = new Request().setHeaders(headers.add(newHeader));
newRequest.setEntity(....newEntity);
handler.handle(context, newRequest).
thenAlways(newRequest::close);
});
return promiseToBeReturned;
}
}
The above code shows that inside the filter there are two requests made. The second request won’t release the resources with the existing ForgeRock code.
Instead use a different handler in the route, which can execute after this filter by using handler.next(…) or If there is no choice to change the route, then a custom HttpClient can be implemented, which can be used to do any of the inner calls and close them to release resources.
Custom HttpClient can be used like Apache Http Client, set the connection resources like max-routes per connection, total connections, timeouts etc as required and tune them as per performance requirements.
Logging Level:
Use Error Level Logging for production use cases, don’t use Debug or Info level as they can cause performance implications.
Loggers:
Do not use system.ou.println(…) for logging for debug or error as it won’t release the resources and can cause thread locks.
No of workers:
Generally should be equal to the number of CPU Cores of the VM in which it is running, can be optionally tuned for routes having more frequent calls.
Additional JVM settings :
As IG requests and response generally have more data that is common like header names, request and response body keys, using string deduplication can also help : -XX:+UseStringDeduplication
Other
JVM Settings and Cache :
(XMX and XMS) is based on a calculation that considers the size of keys and values and the cache size. Performance Tuning :: ForgeRock Directory Services
Based on the deployment approach, whether to use shared cache or cache per backend and the above calculation, recheck Xmx and Xms options.
Using G1GC, try to limit the GC pause time for minor GCs to 200ms, -XX:MaxGCPauseMillis=200
Ulimits
Ulimits for number of open files need to be appropriately set for the user with which the application runs for all components, general recommendation is to set the soft limit to 65536 and hard limits to 131072 for open files.
For those customers using JBOSS (we don't recommend it), then please take note of the additional performance tuning for IG and AM . Note IG now works in standalone mode (which we do recommend) and AM works well with Tomcat:
Below can be adjusted in the standalone.xml
Default worker Threads
Set the default worker threads as per the load requirements. Ex: default value is 100, which can be set to 1000s, as per the load requirement.
Threads (core, max, queue) :
Adjust the core, max and queue threads to meet the requirements, can be set equal to the default worker threads.
KeepAlive Time out:
Default is 100 milliseconds, which is very low for IG and AM connections. This can be adjusted to 30/60 seconds depending on the request clients to the JBoss server.
Max-connections:
max-connections set in the http(s)-listener, default 20000, which can be removed or increased to allow JBoss to handle more number of concurrent connections.
Below Java Settings can be applied in env.sh and env.properties :
Max Http Connections :
Adjust the apache max http connection threads and count in env.properties :
org.apache.tomcat.util.net.MAX_THREADS=10000 (can be set as 10000s as per requirement)
org.apache.tomcat.util.http.Parameters=100000 (can be set as 100000s as per requirement)
There is a lot that goes into performance tuning, but when done well, ForgeRock is incredibly stable and performant. If you need help with tuning your ForgeRock stack, please don't hesitate to reach out to Midships at sales@midships.io
I hope you found this blog a helpful starting point.
Ravi.