Clear pending XA transactions (Oracle Thin/XA Driver) on Weblogic

By one of our customer, I met the following errors regularly on WLS production environment;

java.sql.SQLException: Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceDeadException: Pool XXXJDBCDataSource is disabled, cannot allocate resources to applications.

 Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource '=XXXJDBCDataSource': XAER_RMERR : A resource manager error has occured in the transaction branch  oracle.jdbc.xa.OracleXAException.Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'XXXJDBCDataSource': XAER_RMERR : A resource manager error has occured in the transaction oracle.jdbc.xa.OracleXAException

To Fix this problem we need to follow the following steps:

First of all I tried to execute the following command;

grant select on v$xatrans$ to public;but I got the following error;

SQL> desc v$xatrans$
ERROR:
ORA-04043: object v$xatrans$ does not exist.

so we will need to execute the following script;

•  Log on to database as system user
• Execute sql script xaview.sql in /ORACLE_HOME/rdbms/admin
• SQL>

select object_name ,object_type from dba_objects where object_name like '%XATRANS%'

 

OBJECT_NAME                                                                                                                      OBJECT_TYPE

------------------

D$PENDING_XATRANS$                                                                                                               VIEW

V$PENDING_XATRANS$                                                                                                               SYNONYM

D$XATRANS$                                                                                                                       VIEW

V$XATRANS$                                                                                                                       SYNONYM

• Execute the following command 
• grant select on v$xatrans$ to public (or );
• Execute sql:
• grant select on pending_trans$ to public;
• Execute sql:
• grant select on dba_2pc_pending to public;

 

Any database account performing distributed transactions must have the following privileges:

 

• Execute sql:

  grant select on dba_pending_transactions to public;

• Execute sql:

  grant force any transaction to public; 

you need to enable cleaning unfinished/pending transactions in Weblogic Server 

• Login to AS console
• Home >Summary of Servers >Summary of JDBC Data Sources > XX DataSource> Transcaction
• Enable Recover Only Once for the JDBC data source.
• Enable XA End Only Once for the JDBC data source

 

 

Reason to enable “Recover Only Once”?

The WebLogic Server transaction manager retries the commit call every minute, until a valid XAResource instance is registered with the WebLogic Server transaction manager, if the setting “Recover only once” is allowed and the commit call failure then the Weblogic Server transaction manager calls recover on the resource only once and not every minute.

For more details please visit the following URL;

http://docs.oracle.com/cd/E13222_01/wls/docs100/jta/jtatxexp.html#wp1053782

Problem with access SOA EM

Problem with access SOA EM






[ERROR] [J2EE JMX-46335] [] [tid: oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Wed Mar 18 08:53:24 CET 2015)] [userId: ] [ecid: d490ace35472fb51:-7e89a104:14c214f7eba:-8000-00000000000a2aea,0] [APP: soa-infra] MBean attribute access denied. [[
  MBean: EMDomain:Name=soa-infra,EMTargetType=oracle_soainfra,type=EMIntegration,Application=soa-infra
  Getter for attribute EMInstanceProperties
  Detail: Access denied. Required roles: Admin, Operator, Monitor, executing subject: principals=[] java.lang.SecurityException: Access denied. Required roles: Admin, Operator, Monitor, executing subject: principals=[]

This can be happened on SOA cluster distributed on different physical servers. Only one server which is running on the same server as Admin Server is able to authenticate the users for SOA application & the other servers are not able to authenticate the users.


Solution:


- login into WLS console
- select the domain -> security -> Embedded LDAP
-  enable Refresh Replica At Starup ( default value is disable)
- save changes & restart all servers

Weblogic Stuck Threads

Weblogic Stuck Threads  (BEA-000337)








WLS generate the following errors;





<[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "622" seconds working on the request "Workmanager: default, Version
: 0, Scheduled=true, Started=true, Started time: 622750 ms
[
POST /soa-infra/services/xxxxxxxxxxx/HTTP/1.1
Content-Type: text/xml; charset=utf-8
SOAPAction: ""
Transfer-Encoding: chunked
User-Agent: Java1.6.0_81
Accept: text/html, image/gif, image/jpeg, */*; q=.2
Connection: Keep-Alive


]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-200 "[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)



In SOA environment most of Stuck Threads are generated by de applications self (services).


The first thing you need to contact the developers to check there service


As a workaround

Make the following modifications:

1. Login to Weblogic Server Administration Console.  Default: http://localhost:7001/console
2. Select 'Lock and Edit' (if WebLogic server is a Production Mode implementation)
3. Go to  Home > Summary of Servers > AdminServer, Tuning tab.
4. Change the Stuck Thread Max Time from 600 to 900.
5. Change the Stuck Thread Timer Interval from 60 to 100
6. Go to Overload tab
7. Change the Max Stuck Thread Time from 600 to 900.
8. Change the Stuck Thread Count 0 to 16.
9. Save and Activate Changes.
10. Restart the WebLogic server.

WebCenter (ADF) & Weblogic

WebCenter (ADF) & Weblogic





One of the challenge which I met by one of our customer was the problem with ADF application & Weblogic. 


Different types of errors like JBO-25009 can not create an object of type .... & JBO-26041: failed to post data during insert or


JBO-29114 ADFContext is not setup to process messages for this exception. Use the exception stack trace and error code to investigate the root cause of this exception. Root cause error code is JBO-29000. Error message parameters are


{0=java.sql.SQLException, 1=Protocol violation: [0]}
 This a well known error in Weblogic. WLS has a feature enabled by default in its JDBC Data SOurce That Wraps JDBC data types and which interferes with some LOB type handling in ADF; some issues in passivation for 10g ADF and other LOB interaction from the Entity Objects .


Solution:


Only from WLS 10.3.4 & late you can enable & disable this feature.


 login on WL console -> Services -> Data Sources-> (data Source Name)-> Connection Pool->


Advanced.
under Advanced check out Wrap Data Types.
By checking out Wrap Data Types this feature will be disabled.
Restart of the data source is needed but I prefer to restart Managed Server

weblogic server failed_not_restartable

You get this error during the start of  AdminServer or even Managed Server. There're a lot of threads about this erro &  how to solve it.

In my Situation, I had problem with the Admin Server & Managed Servers

I got to solve the problem of Admin Server as following
wls:/nm/base_domain> nmServerStatus ('AdminServer')


FAILED_NOT_RESTARTABLE
Solution:
1. An authorized user issues the WLST offline command nmConnect to connect to
the Node Manager process on the machine that hosts the Administration Server.
(If the Node Manager instance is the SSH version, the user can connect using the
SSH client). The nmConnect command provides the Node Manager user name
and password that are used to authenticate the user with the Node Manager.
Then, the user issues the nmStart command and provides the credentials for
starting the Administration Server.
For example:


prps = makePropertiesObject("username=weblogic,password=XXXXX")


prps = makePropertiesObject("username=weblogic, password=welcome1")

nmStart("AdminServer",props=prps)

The nmStart command identifies the domain and server instance to start.

2. Node Manager looks up the domain directory in nodemanager.domains, and authenticates the user credentials using a local file that contains the encrypted username and password.

3. Node Manager obtains the startup properties for the Administration Server.

4. Node Manager creates the Administration Server process.

5. The Administration Server obtains the domain configuration from its config directory.

Note: If the user has previously connected to the Node Manager, a

boot.properties file exists, and the user does not have to supply user name and password.

wls:/nm/base_domain> prps = makePropertiesObject("weblogic.ListenPort=7001")

wls:/nm/base_domain> nmStart('AdminServer',props=prps)

Starting server AdminServer ...

Successfully started server AdminServer ...

wls:/nm/base_domain>
=============================
 
For Managed Server
 
A lot of threads say it has to do with Xrs or Xnohup arguments (The arguments to use when starting this server). I disagree with them . Just have a look at  General Node Manager Configurationhttp://docs.oracle.com/cd/E12839_01/web.1111/e13740/nodemgr_config.htm 
 
I think the main problem with failed_not_restartable is error in the server startup configruation file, the config.xml or in PATH or CLASSPATH  ( ID 1130004.1)
 
In my situation I found in error in the server startup configuration file.I added the wrong Root Directory.
ex. D:\weblogic\Oracle\Middleware\user_projects\domains\my_domain
& it must be D:\weblogic\Oracle\Middleware\user_projects\domains\base_domain
 
 
Cheers,
 

Critical error Server failed. Reason there are 1 nested errors

It took to much time to find the solution for this error "Critical Server failed. Reason: There are 1 nested errors: weblogic.management.ManagementException: Booting as admin server, but servername,xxxxx, does not match the admin server name"

This  error can be accured  in the start of Managed Server through Node Manager
Reasons
1- Wrong parameters  in the Config.xml file
2- Wrong parameters in the Server Start file ( startup.properties)
3- Startup the Managed Server for the first time via Node Manager

Solution:

1- Just check one of the two files above &try; find out the wrong parameters then start your managed Server.
In my case  I addede the wrong the Root Directory " The directory that this server uses as its root directory. This directory must be on the computer that hosts the Node Manager"

2- Try starting the managed server one time via the script startManagedWebLogic outside of using Node Manager for example: WL_HOME\user_projects\domains\base_domain\bin\startManagedWebLogic.cmd (name managed Server) http://localhost:7001 . Then stop the server and this time use Node Manager still using startWebLogic as the script.
Cheers,

Weblogic Admin Console Timeout

on the Weblogic Admin Console will cause user session timeout and will fall back at the login screen.

Solution:

To avoid timeout session you'll need to increase session timeout parameter in weblogic.xml.

This file is located under WL_HOME\wlserver_10.3\server\lib\consoleapp\webapp\WEB-INF.
The default setting is as floowing;

session-descriptor 
(timeout-secs)3600(/timeout-secs)
(invalidation-interval-secs)60(/invalidation-interval-secs)

You'll need to change these two parameters to your wish.
After changes has been taken place, just save the file and restart weblogic server.
 
Cheers,