This case study describes the root cause analysis and resolution of an authentication problem (BEA-000386) with Weblogic Integration Admin server.
Environment specifications
Lessons learned below were identified and documented to prevent such problem in the near future:
Environment specifications
· Java EE server: Oracle Weblogic Integration 9.2 MP2 (WLI)
· JDK: Java version: 1.5.0 SR6, IBM VM
· RDBMS: Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production
· Platform type: Middle tier - Order Provisioning System
Problem description
weblogic.security.SecurityInitializationException: Authentication for user <WL user> denied observed at start-up of the WLI Admin server.
<Jan 21, 2011 7:07:41 AM EST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Jan 21, 2011 7:07:41 AM EST> <Critical> <Security> <BEA-090403> <Authentication for user <WL user> denied>
<Jan 21, 2011 7:07:41 AM EST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user <WL user> denied
weblogic.security.SecurityInitializationException: Authentication for user <WL user> denied
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:947)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1029)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:854)
at weblogic.security.SecurityService.start(SecurityService.java:141)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Gathering and validation of facts
As usual, a Java EE problem investigation requires gathering of technical and non technical facts so we can either derived other facts and/or conclude on the root cause. Before applying a corrective measure, the facts below were verified in order to conclude on the root cause:
· Recent change of the platform? Yes, deployment was performed few hours before the problem
· Sudden crash of the Weblogic admin server and/or managed servers? No (important to determine any corruption type of event)
· Any change done at the WLI database level? Yes, deployment process did include the creation of a new WLI domain and re-use of the existing WLI database schema
· Any change of the affected application / WLI domain at file system level? No, files were only modified/created on the new WLI domain which is not sharing the same file system of the existing WLI domain
Root cause
Based on the above facts, we ruled out a local WLI domain security LDAP corruption problem and moved to investigation at the WLI schema / database level where other facts were gathered.
What we found is that the creation process of the new WLI domain, sharing the same WLI schema of the existing domain, removed one entry within USERS & GROUPMEMBERS tables.
Such data is mandatory and is verified on WLI server start-up as Weblogic is looking for a match between such users and group vs. the local security LDAP data.
Solution
Once the root cause was confirmed, we applied the following resolution procedure:
1) Manually insert the missing data within the USERS & GROUPMEMBERS WLI schema tables:
INSERT INTO USERS (U_NAME, U_PASSWORD, U_DESCRIPTION) values ('<WL user>', '<Encrypted Pwd>', 'This user is the default administrator.');
INSERT INTO GROUPMEMBERS (G_NAME, G_MEMBER) values ('Administrators', '<WL user>');
2) Restart the Admin server
3) Restart the managed servers
4) Log monitoring and successful deployment status did confirm the resolution
Recommendations and best practices
· Take proper backup and monitor critical WLI schema and tables such as USERS & GROUPMEMBERS for any unexpected data update or removal
· From an architecture perspective, consider the creation of a separate WLI schema for any new creation of WLI domain which would provide better flexibility and eliminate the risk of any data overlap
