Troubleshooting

Possibilities for tracing and debugging can be very important when trying to find the reason for problems like the examples below:

• The message is not sent to the customer's web service
• Incoming messages are not received by IFS Applications
• Event is not sent by mail

Note:  When debug is activated the performance will be slower. Do not have it activated during normal operation.

Note: IFS Application is using new, common, logging framework. The most of IFS Connect code has been rewritten to use the new concepts in conjunction to the old one, but there are still places only using the old, IFS Connect own, debugging concept.

Contents

• Overview of the IFS Connect debug concept
• Where to find Error messages
• Logging and Debugging of IFS Connect
  • Debugging of Connect Server
  • Debugging of the Application Server
  • IFS Connect own debugging framework
• Batch Processor troubleshooting

Overview of the IFS Connect debug concept

It is recommend to read about the IFS Connect overview before you start to use the debug concept.

1. Ability to debug all message executions that occur in the IFS Connect JEE application.
   Example of message executions:
   - A message is received from Connect Server and is stored to a message queue
   - Batch Processor starts to execute a message located on queue. The message includes a BizAPI call.
2. Ability to debug the readers in the Connect Server. For example the file reader which polls a directory for files.
3. The message traffic to/from the SOAP Gateway can be traced.
    

Where to find Error messages

On Application Server Tasks

Only Application Server Tasks of type Other can be scheduled for background execution. If such tasks fails, the reason for the error will be shown in the Schedule Application Server Tasks window.

From here it is possible to go to the Application Message window for detailed examination by choosing Show Application Message from the context menu.

On Application Messages

If message located on a queue has failed or is in retry mode the error will be shown like below:

Logging and Debugging of IFS Connect

Large part of IFS Connect code is using the new IFS logging framework for logging and debugging. The amount of logged information is controlled by the level of categories 'integration' and 'batchprocessor' (for the core IFS Connect parts) and 'gateway' (for SOAP Gateway). But some parts of IFS Connect framework are still using the old fashion, IFS Connect own, logging framework, so in some cases it can be necessary to work with both concepts.

The installation process creates logging configuration files in the directory <ifs_home>\instance\<instance_name>\conf.
By default there are two configuration files there:

Both configuration files are referred from respective start script/start properties. When creating a new instances of Connect Server you have to create new start script by copying and modifying the existing one – don’t forget to copy and modify the corresponding configuration file for logging and reflect the new file name in the start script!

By default both files create their log files in subdirectories j2ee and connectserver of the logging top directory <ifs_home>\instance\<instance_name>\logs.
Each configuration creates a log file with name ifsalert.log that is supposed to collect all error messages. There will be also an IFS Connect log file with name ifsconnectlog.log (for Middleware Server) or ifsconnect.log (for Connect Server). Logging level for this log file is set to 'info' (read more about logging levels here).

Debugging of Connect Server

In the configuration files there are some commented sections that can be used as templates when enabling debugging of Connector Readers. Those sections look as follow:

handler.<handler name>.type=textfile
handler.<handler name>.filename=<path to the debug file>
handler.<handler name>.filter=ifs.fnd.log.TagFilter,<Reader Name>
handler.<handler name>.includepackage=false

IFS Connect is using the tag concept for marking debug messages with queue and connector names. So simply use the connector name as <Reader Name> to filter messages related to the actual connector. Of course, the logging level has to be set to debug or at least trace. Either globally or for the ‘integration’ category, if you want to exclude some statements from the framework.

A tip: to fetch all other messages, not already handled by any of the definition according to the above description, create yet another Handler according to the following definition:

handler.others.type=textfile
handler.others.filename=<path to the debug file>
handler.others.filter=!ifs.fnd.log.TagFilter,<list of readers>
handler.others.includepackage=false

where <list of readers> is a comma separated list of connectors already handled by the other Handlers.
Note the exclamation mark before the filter specification!

Example: Say your connectserver1-logging.properties has the following contents:

global.level=DEBUG
integration.level=INFO

handler.alert.type=textfile
handler.alert.filename=@IFS_HOME@/instance/@IFS_INSTANCE@/logs/connectserver/ifsalert.log
handler.alert.level=error

handler.connectlog.type=textfile
handler.connectlog.filename=@IFS_HOME@/instance/@IFS_INSTANCE@/logs/connectserver/ifsconnect.log
handler.connectlog.level=info
handler.connectlog.categories=integration
handler.connectlog.includepackage=false

handler.filereader1.type=textfile
handler.filereader1.filename=@IFS_HOME@/instance/@IFS_INSTANCE@/logs/connectserver/filereader1.log
handler.filereader1.filter=ifs.fnd.log.TagFilter,FILE_READER1
handler.filereader1.includepackage=false

handler.mailreader1.type=textfile
handler.mailreader1.filename=@IFS_HOME@/instance/@IFS_INSTANCE@/logs/connectserver/mailreader1.log
handler.mailreader1.filter=ifs.fnd.log.TagFilter,MAIL_READER1
handler.mailreader1.includepackage=false

So now, beside of the alert and log files, you will also get two additional debug files for readers FILE_READER1 and MAIL_READER1.
But if you now want to fetch all the others debug messages in a separate file you can add the following definition:

handler.others.type=textfile
handler.others.filename=@IFS_HOME@/instance/@IFS_INSTANCE@/logs/connectserver/others.log
handler.others.filter=!ifs.fnd.log.TagFilter,FILE_READER1,MAIL_READER1
handler.others.includepackage=false

As mentioned, in some situations it can be necessary to deal with the old, IFS Connect own, debugging framework. The sections below describe how to set up debugging Connect Server.
 

Debugging of the Application Server

The situation with debugging from the Application Server is a bit different. Messages from Batch Processor flow are tagged by queue name and tag type QUEUE. Furthermore logic within the Application Server will execute continuously in one thread only.

So for better overview of debugging printouts you can use a handler definition that will split the output to separate files; one per thread:

handler.connectdebug.type=textfile
handler.connectdebug.filename=<path to the debug file>
handler.connectdebug.categories=integration,batchprocessor,gateway
handler.connectdebug.perthread=true

Middleware Server reuses threads, so one file will most probably contain more than only messages coming from one, single request. But on the other hand the request debugging will be consistent and continuous without interruptions from other threads and requests.

Separate log/debug files can be created on message queue basis using the tag filter in a similar way as described above filtering per Connector Reader in case of the Connect Server logging. There are already some commented template sections for the default queues in the j2ee-logging.properties file. But because Batch Processor also uses Tag Type concept, this is also possible to use the QUEUE tag type to filter messages related to all queues. The commented template section will create log files in a subdirectory message_queues for each queue separately. Example file name: OUT1-ManagedServer1.log 

Typically the log level can be changed in the logging.properties file, which normally requires restart of the actual server or application, or using a JMX console (e.g. Jconsole). But there is one exception from this rule: log level for Message Queues can be set in runtime in IFS Solution Manager > Integration > IFS Connect > Setup IFS Connect and will not require restart.

IFS Connect own debugging framework

As mentioned previously, in some situations it can be necessary to work, in conjunction to the described above common logging framework, also with the old, IFS Connect own, debugging framework. Activation and configuration of IFS Connect own debugging can be done in IFS Solution Manager > Integration > IFS Connect > Setup IFS Connect. Currently only Connector Readers, with exception of JMS reader, uses the old framework. On those readers it is possible to enable/disable debugging, define the debug level and define debug file:

• DEBUG - Use to enable or disable debug information generation.
• DEBUG_LEVEL - Controls amount of debug information.
• DEBUG_FILE - Specifies path to file to which the debug information will be written.

In general all information send to IFS Connect own log files will also be collected by Loggers of the common logging framework, so the files specified by the DEBUG_FILE parameter can in most cases be ignored, but it can be necessary to set up log levels in IFS Connect anyway.

Beside of that Connect Server defines its own LOG_FILE.

IFS Connect uses object (configuration parameter instance) names, i.e. names of readers, senders, queues and servers, as Tags, which can be used in conjunction to the TagFilter to filter logging output relevant for a particular instance. The most of code is already using the new logging framework, so dealing with the old fashion debug parameters should be chosen as a last choice.

Setting up Debugging for Connect Server

To debug Connect Server you have to enable debugging on at least one of connect readers:

By default Connect Server creates its log files, one per reader, in directory <ifs_home>\instance\<instance_name>\logs\connectserver.

Batch Processor troubleshooting

Application flow that involves Batch Processor typically starts in the database and end with sending a message to an external system. An error can occur in any stage of the flow execution and may require different approach to surround the problem. Message processing can involve following steps:

• PLSQL Access Provider creates an Application Message.
  A suitable method in PLSQL Access Provider is called from PL/SQL Business logic. A new Application Message record in state Released is then created and inserted to the FNDCN_APPLICATION_MESSAGE_TAB table. A database trigger on this table, after changing the record state to Processing (if not an InOrder queue) is sending a JMS message using the Oracle Advanced Queuing framework. An error that occurs during this process (in PLSQL AP or trigger) will cause the entire transaction initiated by the business logic to be rolled back and no JMS be sent.
• Oracle Advanced Queuing framework is storing JMS messages in the database table BATCH_PROC_QUEUE_TAB. In the very rare case of failure during message processing by the AQ framework (de-queuing of the message), the message will be stuck in in the table, the Q_NAME column will be set to AQ$_BATCH_PROC_QUEUE_E (normally the table is empty). Application Message will remain in state Processing (Released for InOrder queue).
• Runtime Exception that occurs during message processing by the Bach Processor's Message Driven Bean will cause the entire transaction to be rolled back and the JMS message in the WebLogic Server marked for redelivery. If the processing will continue failing after a number of attempts, the JMS message will be suspended in the JMS queue in WebLogic and can be viewed/deleted using admin console. Error message should be written to the alert.log file.
• An error that occurs during the message processing by the Batch Processor code in the Middleware Server will cause the Application Message to be set to state Failed with proper error message saved on the message.

Normally there should never be any messages in state Released in non InOrder queues (other then ERROR or TRASHCAN). Released messages in non InOrder queues indicates that the database trigger is disabled. Run App_Message_Processing_API.Enable_Triggers to enable triggers and then App_Message_Processing_API.Rerelease_Messages to cause trigger to start processing released messages.

For non InOrder queues there can be a large number of Processing messages, while there should be no more then one Processing message in an InOrder queue. It can take some time to process all the messages, but if messages are note processed it can indicate that the Batch Processor (the batchprocessor-mdb.jar module of ifsapp.ear application) is disabled. If this is the case, reconfigure installation and see to mark Integration Services to deployment.