Java Logging with Nested Diagnostic Context (NDC)

1. Overview

Nested Diagnostic Context (NDC) is a mechanism to help distinguish interleaved log messages from different sources. NDC does this by providing the ability to add distinctive contextual information to each log entry.

In this article, we will explore the use of NDC and its usage/support in various Java logging frameworks.

2. Diagnostic Contexts

In a typical multi-threaded application like a web application or REST APIs, each client request is served by a different thread. The logs generated from such an application will be a mix of all client requests and sources. This makes it difficult to make any business sense of the logs or to debug.

Nested Diagnostic Context (NDC) manages a stack of contextual information, on a per thread basis. The data in NDC is available to every log request in the code and can be configured to log with every log message – even at places where the data is not in scope. This contextual information in each log message helps to distinguish the logs by its source and context.

The Mapped Diagnostic Context (MDC) also manages information on a per-thread basis, but as a map.

3. The NDC Stack in a Sample Application

To demonstrate the usage of an NDC stack, let’s take an example of a REST API that sends money to an investment account.

The information required as input is represented in an Investment class:

public class Investment {
    private String transactionId;
    private String owner;
    private Long amount;

    public Investment (String transactionId, String owner, Long amount) {
        this.transactionId = transactionId;
        this.owner = owner;
        this.amount = amount;
    }
    
    // standard getters and setters
}

The transfer to the investment account is performed using InvestmentService. The full source code for these classes can be found in this github project.

In the sample application, the data transactionId and owner are placed in the NDC stack, in the thread that is processing a given request. This data is available in every log message in that thread. This way, each unique transaction can be traced, and the relevant context of each log message can be identified.

4. NDC in Log4j

Log4j provides a class called NDC which provides static methods to manage data in the NDC stack. Basic usage:

• When entering a context, use NDC.push() to add context data in the current thread
• When leaving the context, use NDC.pop() to take out the context data
• When exiting the thread, call NDC.remove() to remove diagnostic context for the thread and ensure memory is freed (as of Log4j 1.3, no longer necessary)

In the sample application, let’s use NDC to add/remove contextual data at relevant places in the code:

import org.apache.log4j.NDC;

@RestController
public class Log4JController {
    @Autowired
    @Qualifier("Log4JInvestmentService")
    private InvestmentService log4jBusinessService;

    @RequestMapping(
      value = "/ndc/log4j", 
      method = RequestMethod.POST)
    public ResponseEntity<Investment> postPayment(
      @RequestBody Investment investment) {
        
        NDC.push("tx.id=" + investment.getTransactionId());
        NDC.push("tx.owner=" + investment.getOwner());

        log4jBusinessService.transfer(investment.getAmount());

        NDC.pop();
        NDC.pop();

        NDC.remove();

        return 
          new ResponseEntity<Investment>(investment, HttpStatus.OK);
    }
}

The contents of NDC can be displayed in log messages by using %x option in the ConversionPattern used by appender in log4j.properties:

log4j.appender.consoleAppender.layout.ConversionPattern 
  = %-4r [%t] %5p %c{1} - %m - [%x]%n

Let’s deploy the REST API to tomcat. Sample request:

POST /logging-service/ndc/log4j
{
  "transactionId": "4",
  "owner": "Marc",
  "amount": 2000
}

We can see the diagnostic context information in the log output:

48569 [http-nio-8080-exec-3]  INFO Log4JInvestmentService 
  - Preparing to transfer 2000$. 
  - [tx.id=4 tx.owner=Marc]
49231 [http-nio-8080-exec-4]  INFO Log4JInvestmentService 
  - Preparing to transfer 1500$. 
  - [tx.id=6 tx.owner=Samantha]
49334 [http-nio-8080-exec-3]  INFO Log4JInvestmentService 
  - Has transfer of 2000$ completed successfully ? true. 
  - [tx.id=4 tx.owner=Marc] 
50023 [http-nio-8080-exec-4]  INFO Log4JInvestmentService 
  - Has transfer of 1500$ completed successfully ? true. 
  - [tx.id=6 tx.owner=Samantha]
...

5. NDC in Log4j 2

NDC in Log4j 2 is called as Thread Context Stack:

import org.apache.logging.log4j.ThreadContext;

@RestController
public class Log4J2Controller {
    @Autowired
    @Qualifier("Log4J2InvestmentService")
    private InvestmentService log4j2BusinessService;

    @RequestMapping(
      value = "/ndc/log4j2", 
      method = RequestMethod.POST)
    public ResponseEntity<Investment> postPayment(
      @RequestBody Investment investment) {
        
        ThreadContext.push("tx.id=" + investment.getTransactionId());
        ThreadContext.push("tx.owner=" + investment.getOwner());

        log4j2BusinessService.transfer(investment.getAmount());

        ThreadContext.pop();
        ThreadContext.pop();

        ThreadContext.clearAll();

        return 
          new ResponseEntity<Investment>(investment, HttpStatus.OK);
    }
}

Just as with Log4j, let’s use the %x option in the Log4j 2 configuration file log4j2.xml:

<Configuration status="INFO">
    <Appenders>
        <Console name="stdout" target="SYSTEM_OUT">
            <PatternLayout
              pattern="%-4r [%t] %5p %c{1} - %m -%x%n" />
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="com.baeldung.log4j2" level="TRACE" />
            <AsyncRoot level="DEBUG">
            <AppenderRef ref="stdout" />
        </AsyncRoot>
    </Loggers>
</Configuration>

Log output:

204724 [http-nio-8080-exec-1]  INFO Log4J2InvestmentService 
  - Preparing to transfer 1500$. 
  - [tx.id=6, tx.owner=Samantha]
205455 [http-nio-8080-exec-2]  INFO Log4J2InvestmentService 
  - Preparing to transfer 2000$. 
  - [tx.id=4, tx.owner=Marc]
205525 [http-nio-8080-exec-1]  INFO Log4J2InvestmentService 
  - Has transfer of 1500$ completed successfully ? false. 
  - [tx.id=6, tx.owner=Samantha]
206064 [http-nio-8080-exec-2]  INFO Log4J2InvestmentService 
  - Has transfer of 2000$ completed successfully ? true. 
  - [tx.id=4, tx.owner=Marc]
...

6. NDC in Logging Facades (JBoss Logging)

Logging facades like SLF4J provide integration with various logging frameworks. NDC is not supported in SLF4J (but included in slf4j-ext module). JBoss Logging is a logging bridge, just like SLF4J. NDC is supported in JBoss Logging.

By default, JBoss Logging will search the ClassLoader for the availability of back-ends/providers in the following order of precedence: JBoss LogManager, Log4j 2, Log4j, SLF4J and JDK Logging.

JBoss LogManager as the logging provider is typically used inside WildFly application server. In our case, the JBoss logging bridge will pick the next in order of precedence (which is Log4j 2) as the logging provider.

Let us begin by adding the required dependency in pom.xml:

<dependency>
    <groupId>org.jboss.logging</groupId>
    <artifactId>jboss-logging</artifactId>
    <version>3.3.0.Final</version>
</dependency>

The latest version of the dependency can be checked here.

Let’s add contextual information to the NDC stack:

import org.jboss.logging.NDC;

@RestController
public class JBossLoggingController {
    @Autowired
    @Qualifier("JBossLoggingInvestmentService")
    private InvestmentService jbossLoggingBusinessService;

    @RequestMapping(
      value = "/ndc/jboss-logging", 
      method = RequestMethod.POST)
    public ResponseEntity<Investment> postPayment(
      @RequestBody Investment investment) {
        
        NDC.push("tx.id=" + investment.getTransactionId());
        NDC.push("tx.owner=" + investment.getOwner());

        jbossLoggingBusinessService.transfer(investment.getAmount());

        NDC.pop();
        NDC.pop();

        NDC.clear();

        return 
          new ResponseEntity<Investment>(investment, HttpStatus.OK);
    }
}

Log output:

17045 [http-nio-8080-exec-1]  INFO JBossLoggingInvestmentService 
  - Preparing to transfer 1,500$. 
  - [tx.id=6, tx.owner=Samantha]
17725 [http-nio-8080-exec-1]  INFO JBossLoggingInvestmentService 
  - Has transfer of 1,500$ completed successfully ? true. 
  - [tx.id=6, tx.owner=Samantha]
18257 [http-nio-8080-exec-2]  INFO JBossLoggingInvestmentService 
  - Preparing to transfer 2,000$. 
  - [tx.id=4, tx.owner=Marc]
18904 [http-nio-8080-exec-2]  INFO JBossLoggingInvestmentService 
  - Has transfer of 2,000$ completed successfully ? true. 
  - [tx.id=4, tx.owner=Marc]
...

7. Conclusion

We have seen how diagnostic context helps in correlating logs in a meaningful way – from a business standpoint as well as for debugging purposes. It is an invaluable technique to enrich logging, especially in multi-threaded applications.