Common PeopleSoft Integration Broker Issues and Solutions

Integration Broker:  Cannot find message Message Monitor

Possible causes:

1. User does not have security for the message channel.
2. The message monitor criteria have filtered out the message.

Integration Broker:  Messages processed in incorrect order

Possible cause:

1. The Channel has been partitioned, and the resulting subchannels do not match what was assumed for the ordering of the messages.

Integration Broker: Message Instance not created

Possible cause:
1. Message is inactive.

Integration Broker: Message Instance stays in NEW status.

Possible cause:
1. The pub/sub servers are not booted.
2. The Message Dispatcher has crashed or has been brought down.
3. The Item is not at the top of the queue.  All messages with the same Channel/ Subchannel are in the same queue
4. Run appmsgpurgeall.dms
5. Restart PUBSUB processes
6. Force cleanup from the monitor message -> domain status
7. Test your message by routing it through another channel to make sure that it is not an issue with the channel.

Integration Broker: Message Instance stays in STARTED status.

Possible cause:

1. All Message Handlers have crashed or have been brought down. Processing will resume when Message handlers come brought back up 

2. The Message dispatcher processing the message is on another machine, and either the machine or the application server domain is down. 

Integration Broker: Message Instance stays in WORKING status.

Possible cause:

1. Message Handler has crashed. 

2. The Message Handler processing the message is on another machine, and either the machine or the application server domain is down.  The Message handler working on the message is "blocked". The service will timeout, and the Message Dispatcher will retry the message.

Publication Contract not created

Possible cause:

1. Channel routing rules not set up properly. 

Publication Contract stays in NEW status 

Possible cause:

1. Publication Dispatcher has crashed or has been brought down. 

2. Message Channel paused. 

3. Node paused 

4. Previous message had a status of RETRY, ERROR, or TIMEOUT.  

5. The Publication Contract is not at the top of the queue.  All Publication Contracts with the same Channel/ Subchannel and subscribing node are in the same queue

Publication Contract stays in RETRY status

Possible cause:
1. The remote node cannot be "pinged" successfully.  The publication contract will be processed when the remote node comes back up.
2. No publication handler available, either because it's crashed or it has been brought down.

Publication Contract stays in WORKING status

Possible cause:
1. The publication handler processing the contract is on another machine and either the machine or the domain is down. Processing should continue when the pub/sub system on the other machine comes back up.

Subscription Contract not created.

Possible causes:
1. Message Subscription is inactive.
2. Channel routing rules not set up properly.

Subscription contract - Local Node Issue

For a message published by the local node, subscription contract not created for local node even when Message Subscription is active and "Invoke for local publication" is checked.

Possible cause:
1. The local node was not included in the routing rules for the channel.  To subscribe to its own publications, the local node must be included in the channel routing rules.

Subscription Contract stays in NEW status.

Possible causes:
1. The Subscription Dispatcher has crashed or has been brought down.
2. Message Channel Node or System paused.
3. Message Definition not Active.
4. Previous message had a status of RETRY, ERROR, or TIMEOUT.
5. The Subscription contract is not at the top of the queue.  All Subscription Contracts with the same Channel/ Subchannel and subscription owner are in the same queue.

All publication Contracts remain in a NEW status.

Resolution:  The queue was blocked from a Publication Contact that was in a status of ERROR from two weeks prior.  User did not see the error on the message monitor because he was only looking at today.  User entered xx day in and the ERROR publication appeared.  Canceled the error and all subsequent publication contracts processed successfully.   

Subscription Contract stays in STARTED status.

Possible causes:
1. The Subscription Handler has crashed or has been brought down.

Subscription Contract stays in WORKING status.

Possible causes:

1. The Subscription Handler has crashed or has been brought down. Check in Application Messaging Gateway Administrator
2. Also look for any errors in App Serv Log

Subscription Contract in ERROR status. 

 
Possible causes:
1. Subscription PeopleCode errors
2. Application data errors.

Subscription Contract in TIMEOUT status. 

Possible causes:

1.  Subscription PeopleCode errors
2.  If the message works sometimes, and sometimes does not this may be a problem with the application server configuration.  Tune up your application server min/max values, or reconfigure your domain to a medium or large domain.  Also, changing the recycle count for these services from 0 to 25,000 or 10,000 may eliminate this problem.

Integration Broker:  Unable to ping a node. 

Possible causes:

1. The web server for the Gateway is down.
2. The Gateway is not configured properly.
3. The app server for the node is down.
4. Verify url is correct. Copy url in browser address, should see "PeopleSoft <tools release> Application Messaging gateway".
       (note: be sure that url in the manage lookup contains // before an ip address or machine name)
NOTE: In PeopleBooks it is stated as "//http://HRMS-01/servlet/psft.pt8.gateway.GatewayServlet"
            it should read "http://HRMS-01/servlets/gateway"  notice the "//" are missing from the begining             and is should read "servlets" not "servlet"

Access denied, source node not found

Message even when source node defined on target database.
 
Possible cause:
1.  Known bug: Message Node caching not working.  If a new node is added, reboot appserver.

Integration Broker:  "Bad Gateway, General Exception error .." message when pinging node.

Possible Cause:
1. Check Oprid/ Password of node on Configuration Servlet page. (http:// ?/servlet/gateway.administration)

Access denied. No password defined on source message node.

Possible Cause:
 
In 8.1x Source Message Node properties has entries in 'Password' and 'Re- verify Password'.
Solution: Blank these fileds and ping the source node it worked fine.


Page cannot be displayed message when attempting to bring up Gateway or gateway Administration page. 
 
Possible Causes:
 
3. Make sure Apache/ Jserv started.
4. Make sure aliases are correct in zone.properties

Ping failed, Application server log message saying cannot find JVM.

Resolution:  The application server was on a different machine as the web server.  The Sun JRE 1.2 was downloaded on the web server machine.  Sun JRE 1.2 was re-downloaded on the application server machine and the setting JavaVM Shared Library in psappsrv.cfg file was set to point to the location of this JRE.

All attempted publishing messages from PIA resulted in the following error message

Resolution:  When the upgrade was performed to the production database the following people tool tables had no data in them. 
PSAPMSGPUBINST  (channel names and latest pubid)
PSAPMSGPUBCSYNC (channel names and update timestamp)
PSAPMSGPUBSYNC (channel names and update timestamp)
PSAPMSGSUBCSYNC (channel names and update timestamp)


Additional Notes:

1. Problem: Message instance/Publication Contract sits in NEW status for over 15 minutes with no other messages in progress.
 
Possible Causes:
I. Incorrect Gateway URL defined on Message Node.
II. Message nodes not used in any of the messages should not contain an URL. 
Example:  PSFT_CR which does not have an URL defined.
Actions:  Check URL on Gateway and ping message node.

2. Problem: Publication Contract sits in NEW status status for over 15 minutes with no other messages in progress.
 
Possible Causes:
 
I. Publishing Application Server down.
II. Publishing Pub/Sub processes not configured on App Server domain.
III. Message Channel Paused.
IV. Incorrect Gateway URL defined on Message Node.
V. Node paused.
VI. Previous Message had errors or timed out.
VII. No Publication PeopleCode exists, or PeopleCode is incorrect.
Actions:  Check listed system components, then resubmit message.

3. Problem: Publication Contract sits in WORKING status for over 15 minutes with no other messages in progress.
 
Possible Causes:
 
I. Single threading on the application server is making things slow…if the Pub/Sub processes are running
II. Pub/Sub processes not responding
Actions:  Add more handlers and reboot app server.

4. Problem: Publication Contract is in TIMEOUT status
 
Possible Causes:
 
I. Subscribing Application Sever down.
II. Subscribing Node is Paused.
III. Message Node URL is incorrect.
IV. No Subscription PeopleCode exists on Subscribing system, or Subscription PeopleCode is incorrect.
Actions:  Check listed system components, then resubmit message.

5. Problem: Publication Contract is in ERROR status
 
Possible Causes:
I. Message Node URL is incorrect.
II. Subscription PeopleCode is bad.
Actions:  Check listed system components, if fix is made then resubmit message.  Otherwise go to Resolution of Messages in Error status section.

6. Problem: Publication Contract is in RETRY status
 
Possible Causes:
III. “PS” Password changed on Gateway
Actions:  Check  ~/Gateway.administration and apply password updates to PS ID.

Subscription Process

1. Problem: Subscription Contract sits in NEW status for over 15 minutes with no other messages in progress.
 
Possible Causes:
 
I. Application Server down.
II. Pub/Sub processes not configured on App Server domain.
III. Message definition not active.
IV. Message channel paused.
V. Node paused.
VI. Previous Message had errors or timed out.
VII. If you’re missing a row in PSAPMSGSUBPRCID, do the following:
insert into PSAPMSGSUBPRCID values(0)  [This should never happen]
Actions:  Check listed system components.  Reboot Pub/Sub services if necessary.

2. Problem: Subscription Contract sits in STARTED status status for over 15 minutes with no other messages in progress.
 
Possible Causes:
 
I. Subscription Handler down
Actions:  Check listed system components, then resubmit message.

3. Problem: Subscription Contract sits in WORKING status status for over 15 minutes with no other messages in progress.
 
Possible Causes:
 
I. No Subscription PeopleCode exists
II. Subscription Handler crashed while processing message.
Actions:  Check listed system components.   Reboot Pub/Sub services if necessary.

4.Problem: Subscription Contract in ERROR status
 
Possible Causes:
 
I. Subscription PeopleCode errors.
II. Application Data errors.
Actions:  First resubmit message.  If it fails then go to” Resolution of Messages in Error status” section (below.)

5. Problem: Subscription Contract is in EDIT status

Possible Causes:
I. Message was edited and has yet to be resubmitted for processing.
Actions:  Normally this status only applies when an error is being reconciled by support personnel.

6. Problem: Subscription Contract is in RETRY status

Possible Causes:
I. Subscription PeopleCode is empty.
Actions:  This applies when a message has been resubmitted.

7. Problem: What happens when your message is routed to Both  (Publish & Subscribe) and the message instance is DONE, but  no subscription contract is created?   
Possible Causes:
I. Check your message subscription properties to make sure that the Local Publication ‘Invoke for local publication’ is turned on.
Actions:  This should never happen.  Escalate to IS support.

 

Other Possible Problems

1. Problem: Can’t see Messages in Message Monitor

Possible Causes:  
I. Channel Security not set.
II. Message Monitor view criteria is too limiting (looking at the wrong time, channel, node, etc.)
Actions:  Take corrective measures.

2. Problem: My Publish() PeopleCode finishes successfully, but there is no message in the Message Monitor.
Possible Causes:
I. The Message Definition is InActive.
Actions:  Take corrective measures.

3. Problem: Message Instance sits in NEW status.
Possible Causes:
1.   Application Server down.

II. Pub/Sub services not configured on Application Server domain.
III. Message Broker Dispatcher down.
Actions:  Check pub/sub services and reboot them if necessary.

4. Problem: Message Instance sits in WORKING status.
Possible Causes:
I. Message Broker Handler crashed.
Actions:  Take corrective measures.
Resolution of Messages with Error status
1. From Sub Contracts Page, click on details for message in Error status.
2. Next, click on the Message Errors tab and check the Error Message
3. If the Error Message does not imply a solution then check on the app server log file for error message details, at /ps/c1prd/appserv/C1PRD/LOGS/APPSRV.LOG
For example:
· PSSUBHND_dflt.1169 [08/15/01 08:00:14 SubConProcess](1) GenMessageBox(6540, 11, M): Component Processor: Warning -- Address should consist of at least Street and City. (6540,11)
· PSSUBHND_dflt.1169 [08/15/01 08:01:25 SubConProcess](1) GenMessageBox(200, 0, M): /ds3/home/build81/pt81410rc6-retailr_o/src/psppr/ramget.cpp: SQL error. Stmt #: 609  Error Position: 0  Return: 8118 - Memory allocation failed (SQLSTATE PS118)
· PSSUBHND_dflt.1169 [08/15/01 08:01:25 SubConProcess](1) GenMessageBox(200, 0, M): SQL Access Manager: SELECT COUNTRY, DESCR, ADDR1_LBL, ADDR1_AVAIL, ADDR2_LBL, ADDR3_LBL, ADDR4_LBL, CITY_LBL, CITY_AVAIL, NUM1_LBL, NUM2_LBL, HOUSE_TYPE_LBL, ADDR_FIELD1_LBL, ADDR_FIELD2_LBL, ADDR_FIELD3_LBL, COUNTY_LBL, STATE_LBL, POSTAL_LBL, POST_SRCH_AVAIL, GBSYS_NRPATH_UK, GBSYS_CFGPATH_UK FROM PS_COUNTRY_TBL WHERE COUNTRY=:1 ORDER BY COUNTRY
· PSSUBHND_dflt.1169 [08/15/01 08:01:25 SubConProcess](1) GenMessageBox(200, 0, M): /ds3/home/build81/pt81410rc6-retailr_o/src/psapmsg/logerror.cpp: SQL error. Stmt #: 628  Error Position: 0  Return: 8118 - Memory allocation failed (SQLSTATE PS118)
· PSSUBHND_dflt.1169 [08/15/01 08:01:25 SubConProcess](1) GenMessageBox(200, 0, M): SQL Access Manager: SELECT MAX(SEQNO) FROM PSAPMSGSUBCERR WHERE PUBNODE = :1 AND CHNLNAME = :2 AND PUBID = :3 AND MSGNAME = :4 AND SUBNAME = :5
4. In this case, we resubmitted the message (see step#1 for Resubmit button) and it later successfully processed.
SQL and PC Trace options for App Server - good for checking if SQL events didn't take place or if Subscription PC didn't fire properly.

SQL Tracing on App Serv :In the AppServer Configuration file you can set the trace options.

Trace
After you have set the Domain Settings the Trace section appears. This section allows you to specify the tracing options that you can enable on the application server to track the SQL and PeopleCode of your domain(s).
Values for config section - Trace
    TraceSql=0
    TraceSqlMask=12319
    TracePC=0
    TracePCMask=4095
    TracePPR=0
    TracePPRMask=4095
    TraceAE=0
    Write crash dump to separate file=Y
Do you want to change any values (y/n)? [n]:
TraceSQL
Sets logging level for SQL tracing for all clients. Traces are written to the following location: <PS_HOME>/appserv/<domain>/LOGS/<DomainOPRID>_<svrname>.tracesql.
If you enter 0 it disables tracing; use 7 to enable a modest tracing level for debugging. For other levels of tracing, set to a value equal to the sum of the desired options. For example, if you only want to trace SQL, TraceSQL=1; if SQL statements and Connect statements are desired, TraceSQL should be set to 1+ 2 + 4 = 7. A setting of TraceSQL = 7 is recommended for troubleshooting connection and other basic problems. Tracing can consume large amounts of disk space over time so be sure to reset TraceSQL = 0 when you've finished trouble shooting.
TraceSQLMask
Sets logging level ceiling for SQL tracing for individual clients. Traces are written to the following location: <PS_HOME>/appserv/<domain>/LOGS/<ClientOPRID>_<svrname>.tracesql. Clients must specify desired SQL tracing level using PeopleSoft Configuration Manager on the Trace tab. To prevent clients from turning on the application server trace, and consuming resources, the application server uses TraceSQLMask as an administrative control facility.
If a client transmits a request to trace SQL, the application server compares the value transmitted to TraceSQLMask. If the client value is less than or equal to TraceSQLMask, the application server enables the trace. However, if the client value is greater, application server will enable the trace up to the TraceSQLMask value. Trace files are written on the application server; no trace shows up on the client workstation.
Trace values are set in the application server configuration file psappsrv.cfg file. Output files are written to directory $PS_HOME/appserver/winx86/<domain>/logs.
For TraceSQL and TraceSQLMask you have the following options:
1         - SQL statements
2         - SQL statement variables
4         - SQL connect, disconnect, commit and rollback
8         - Row Fetch (indicates that it occurred, not data)
16        - All other API calls except ssb
32        - Set Select Buffers (identifies the attributes of columns
            to be selected).
64        - Database API specific calls
128       - COBOL statement timings
256       - Sybase Bind information
512       - Sybase Fetch information
4096      - Manager information
8192      - Message Agent information
TracePC

Sets a desired level for PeopleCode tracing for activity generated by all clients on a domain. Eligible values will be defined in the configuration file. TracePC values are displayed in the Configuration Manager on the Trace tab. You can find the results in the following location: <PS_HOME>/appserv/<domain>/LOGS/<domain>.log.
TracePCMask

This parameter controls which of the PeopleCode Trace options requested by client machines will be written to the trace file. The results of this trace are written to
<PS_HOME>/appserv/<domain>/LOGS/<ClientMachine>.<domain>.log
For TracePC and TracePCMask you have the following options.
1         - Trace instructions
2         - List the program
4         - Show assignments to variables
8         - Show fetched values
16        - Show stack
64        - Trace start of programs
128       - Trace external function calls
256       - Trace internal function calls
512       - Show parameter values
1024      - Show function return value
2048    - Trace each statement in program
For the TraceSQL I would like for you to use a value of "5" and for the TracePC use a value of "10"

 

Issue – PeopleSoft Integration Broker Messages Stay in New Status

Check if PUBSUB enabled application server is not down.

Verify that PUBSUB is enabled in psadmin.

Check the status of the application server and verify that the following 6 programs are running (Fig. 2): PSBRKDSP.exe, PSBRKHND.exe, PSPUBDSP.exe, PSPUBHND.exe, PSSUBDSP.exe and PSSUBHND.exe

Check the Domain status in Integration Broker -> Service Operations Monitor -> Administration ->  Domain Status : You should have domains which are Active. Update the status if they are inactive by using “All Domains Active”

Check Domain status (as in Fig. 1) – It is possible that only one domain is configured for PUBSUB services and is set as a slave domain in psappsrv.cfg. It is also possible that all the domains are set as slave in case of multiple domains (picture below). A value of 0 will initialize the PUB/SUB servers in normal (Master) mode.

You will need to bounce the appserver if you change the settings of Domain Slave Mode. Also, If domains change the status back to inactive after you manually update them to active – you should restart the pubsub enabled application servers.

Check the Queue status in Integration Broker -> Service Operations Monitor -> Administration -> Queue Status: The status of the queue should not be Paused. If it is paused, run it.

Ping Node to verify that you get the Success (117, 73) message. Do this ping node test for all the nodes, which are not processing the messages, including the default local node.

Check if there are messages stuck in the following status: “Timeout” or “Started” or “Working” – if messages are in this status, it means that they are being submitted again and again to be processed. This causes “New” status queue to increase. Debug the reason why they are stuck in this state or cancel them to allow the new ones to be processed. Make sure that you can work on the cancelled ones later on.

Bounce the PUBSUB enabled appserver if nothing.