Thursday, May 5, 2016

WebLogic Server domain logs old are out of order

Problem:

When inspecting the WebLogic Server domain log, the entries are out of order. Not only that, but sometimes they are days apart!
####<May 2, 2016 12:00:01 AM EDT> <Notice> <Stdout> <machine1> <soa_server1> ...
####<Apr 30, 2016 2:18:42 PM EDT> <Critical> <Health> <machine1> <AdminServer> ...
####<Apr 30, 2016 4:01:29 PM EDT> <Critical> <Health> <machine1> <AdminServer> ...
####<Apr 30, 2016 5:32:44 PM EDT> <Critical> <Health> <machine1> <AdminServer> ...
####<Apr 30, 2016 6:16:14 PM EDT> <Critical> <Health> <machine1> <AdminServer> ...
####<Apr 30, 2016 12:14:45 PM EDT> <Notice> <Stdout> <machine2> <soa_server2> ...
####<Apr 30, 2016 11:46:08 PM EDT> <Notice> <Stdout> <machine2> <soa_server2> ...
####<Apr 30, 2016 11:46:08 PM EDT> <Notice> <Stdout> <machine2> <soa_server2> ...
####<Apr 30, 2016 11:46:08 PM EDT> <Notice> <Stdout> <machine2> <soa_server2> ...
####<Apr 30, 2016 11:46:13 PM EDT> <Notice> <Stdout> <machine2> <soa_server2> ...
####<May 1, 2016 6:36:32 AM EDT> <Notice> <Stdout> <machine2> <soa_server2> ...
####<May 1, 2016 6:36:32 AM EDT> <Notice> <Stdout> <machine2> <soa_server2> ...
####<May 2, 2016 1:22:29 AM EDT> <Notice> <Stdout> <machine2> <soa_server2> ...

Explanation:

The Oracle documentation explains this extremely well:
Each WebLogic Server instance writes all messages from its subsystems and applications to a server log file that is located on the local host computer. By default, the server log file is located in the logs directory below the server instance root directory; for example, DOMAIN_NAME/servers/SERVER_NAME/logs/SERVER_NAME.log, where DOMAIN_NAME is the name of the directory in which you located the domain and SERVER_NAME is the name of the server. 
In addition to writing messages to the server log file, each server instance forwards a subset of its messages to a domain-wide log file. By default, servers forward only messages of severity level NOTICE or higher. While you can modify the set of messages that are forwarded, servers can never forward messages of the DEBUG severity level.
The domain log file provides a central location from which to view the overall status of the domain. The domain log resides in the Administration Server logs directory. The default name and location for the domain log file is DOMAIN_NAME/servers/AdminServer/logs/DOMAIN_NAME.log. 
The timestamp for a record in the domain log is the timestamp of the server where the message originated. Log records in the domain log are not written in the order of their timestamps; the messages are written as soon as they arrive. It may happen that a Managed Server remains out of contact with the Administration Server for some period of time. In that case, the messages are buffered locally and sent to the Administration Server once the servers are reconnected.
In each managed server, under the Logging tab, under Domain log broadcaster, you will notice these modifiable settings. The Severity level indicates at which level (and higher) you want the logs forwarded to the central domain log.


The Buffer Size is recommended to be set to 10 in production mode and 1 in development mode. Is this the number of minutes? Number of hours? Number of lines? Number of kilobytes? Nobody knows.

However, what is important to know is that the messages will be forwarded only when the buffer is full. This could take days, so don't be alarmed if you notice log entries in the domain log that appear to be old.


References:





No comments: