Sunday, July 5, 2015

Provisioning an AWS instance for general Oracle software

This blog post describes how to provision a barebones Linux server from Amazon Web Services (AWS) for the purpose of installing general Oracle software.

This instructions optionally include adding an extra mount point, enabling NFS, and installing VNC for remote GUI connectivity.


Create an EC2 Instance

1. Login to the AWS Management Console at:

https://us-west-2.console.aws.amazon.com/console/home

2. Click on "EC2" on the left-hand menu



3. Navigate to Instances > Launch Instance

4. "Select" the Red Hat Enterprise Linux 7.1 instance type


5. Select an instance type (e.g., t2.medium)


6. Click on "Next: Configure Instance Details"


7. Keep all defaults

8. Click on "Next: Add Storage"


9. Click on "Add New Volume"

10. Enter the size of "10" for 10GB


11. Click on "Next: Tag Instance"

12. Enter the name of your instance (just a name, e.g., "My Oracle Server")


13. Click on "Next: Configure Security Group"


14. Provide a Security Group name and description, and add the firewall rules you want to allow


15. Click on "Review and Launch"



16. Click on "Launch" after reviewing the settings



17. Enter a new key pair name, download key pair, and click on "Launch Instances"



18. Click on "View Instances" and the instance should be up in a few minutes




Create an Elastic IP

19. Click on "Elastic IP" on the left-hand menu

20. Click on "Allocate New Address"


21. Click on "Yes, Allocate"

22. Click on "Close"

23. Right-click on the IP address and select "Associate Address"

24. Click on the Instance field and select your EC2 instance from the list



Create a Putty PPK File for SSH Access

25. Follow these instructions to convert the downloaded .pem file to a Putty compatible .ppk file


26. Login to the server via Putty or equivalent SSH client as the "ec2-user"

27. Then type sudo su - to login as the root user




















Setup the Linux Server

28. Install some basic Oracle required packages

yum install telnet
yum install wget
yum install gcc
yum install gcc-c++
yum install glibc-devel
yum install libaio
yum install libaio-devel
yum install sysstat
yum install libstdc++-devel
yum install compat-libstdc++
yum install compat-libstdc

29. Define a hostname

hostname oradev.raastech.com

30. Type the following to get your local IP address

ifconfig -a | grep broadcast | awk '{print $2}'

31. Edit the local hosts by typing vi /etc/hosts file and manually add your public and private IP addresses and a hostname of your choosing

52.27.XXX.XXX    oradev-ext.raastech.com   oradev-ext
172.31.XXX.XXX   oradev.raastech.com       oradev

32. Create the Oracle unix user

groupadd oinstall
groupadd dba
useradd nobody
useradd -c "Oracle Software Owner" -g oinstall -G dba oracle
passwd oracle

33. Edit the profile for both the 'root' and 'oracle' users to something more readable

vi /root/.bash_profile /home/oracle/.bash_profile

34. Add the following and save the file

export PS1="\u@\h:\$PWD> "
alias ls='ls'
alias ll='ls -l'

35. Edit the sysctl file

vi /etc/sysctl.conf

36. Add the following and save the file, updating the hostname and domainname accordingly

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65535
# Semaphores: semmsl, semmns, semopm, semmni
kernel.shmmni = 4096
kernel.sem = 256 32000 100 142
fs.file-max = 6815744
fs.aio-max-nr = 1048576
kernel.hostname   = oradev.raastech.com
kernel.domainname = raastech.com
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_max=1048576
kernel.msgmni = 2878

37. Load the sysctl settings

sysctl -p

38. Edit the configuration for the pam_limits module

vi /etc/security/limits.conf

39. Add the following to the bottom and save the file

oracle  soft    nofile  4096
oracle  hard    nofile  65536
oracle  soft    nproc   2047
oracle  hard    nproc   16384


Add an Extra 10GB Mount Point (optional)

40. Create a mount point

mkdir -p /u01
chown oracle:oinstall /u01

41. Add the extra EC2 volume and mount it

echo "View available disks"
lsblk

echo "Confirm that /dev/xvdb is a 'data' volume"
file -s /dev/xvdb

echo "Create file system, all data will be lost in it"
mkfs -t ext4 /dev/xvdb

echo "Backup fstab and edit it"
cp /etc/fstab /etc/fstab.orig.20150528
echo "/dev/xvdb       /u01   ext4    defaults,nofail        0       2" >> /etc/fstab

echo "Mount /u01"
mount /u01
chown oracle:oinstall /u01

echo "Confirm that /u01 is mounted and available"
df -m


Install an NFS Server (optional)

42. Install the necessary NFS packages

yum install nfs-utils

43. Create a share folder

mkdir -p /u01/share
chown oracle:oinstall /u01/share

44. Edit the /etc/exports file

vi /etc/exports

45. Add the following and save the file (the hostnames denote which clients should have access)

/u01/share oraclient1.raastech.com(no_root_squash,rw,sync)
/u01/share oraclient2.raastech.com(no_root_squash,rw,sync)

46. Start the NFS server

service rpcbind start
service nfs start

47. Open port 2049 on the local firewall to allow remote NFS access to other EC2 instances

service firewalld start
firewall-cmd --permanent --zone=public --add-port=2049/tcp
firewall-cmd --reload


Configure the NFS Clients (optional)

48. On the client servers, run the following commands

yum install nfs-utils
telnet soadb.raastech.com 2049
mkdir -p /u01/share
chown oracle:oinstall /u01/share
sudo echo "oradev.raastech.com:/u01/share   /u01/share      nfs     defaults        0 0" >> /etc/fstab
mount /u01/share
df -m


Install and Configure the VNC Server (optional)

49. Install the required VNC packages

yum groupinstall 'Server with GUI'yum install tigervnc-server
yum install xclock

50. Open up the VNC port 5901 for remote access

service firewalld start
firewall-cmd --permanent --zone=public --add-port=5901/tcp
firewall-cmd --reload

51. Start the VNC server

vncserver :1 -geometry 1280x720 -depth 16

52. Edit the profile script

vi /root/.bash_profile

53. Add the following and save the file

xhost +

54. Edit the profile script of additional unix users that will require GUI access

vi /home/oracle/.bash_profile

55. Add the following

export DISPLAY=:1

56. Connect via a VNC client such TightVNC on port 1






















Applicable Versions:
  • Amazon Web Services (2015)
  • Red Hat Enterprise Linux 7.1 64-bit


Saturday, July 4, 2015

"Inactive Connection Timeout" and "Remove Infected Connections Enabled" parameters in WebLogic Server

Have you ever wondered what the Inactive Connection Timeout and Remove Infected Connections Enabled settings do in Oracle WebLogic Server 11g/12c? They help address some of the issues your application may be experiencing as it pertains to data sources.

Accessing these parameters is done by simply navigating to Data Sources > [data source name] > Configuration > Connection Pool > Advanced.



Inactive Connection Timeout

Consider this parameter as a short term solution if you suspect your Java code having a connection leak. For example, a connection may be opened in the code but never closed. Usually you will see the number of 'Active Connections' increasing in the WebLogic Admin Console over time. Setting this parameter will force WebLogic Server to release the connection back to the connection pool after X seconds of inactivity.

The Oracle documentation states:
The number of inactive seconds on a reserved connection before WebLogic Server reclaims the connection and releases it back into the connection pool.
You can use the Inactive Connection Timeout feature to reclaim leaked connections - connections that were not explicitly closed by the application. Note that this feature is not intended to be used in place of properly closing connections.
When set to 0, the feature is disabled.
When this parameter is triggered, the following will appear in the logs:
####<Jul 1, 2015 8:30:47 AM EDT> <Warning> <JDBC> <soahost1> <soa_server1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1435753847302> <BEA-001153> <Forcibly releasing inactive/harvested connection "[weblogic.jdbc.wrapper.JTAConnection_weblogic_jdbc_wrapper_XAConnection_oracle_jdbc_driver_LogicalConnection-SOADataSource-9054, oracle.jdbc.driver.LogicalConnection@1a72fd3f]" back into the data source connection pool "SOADataSource", currently reserved by: java.lang.Exception        at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:356)        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:364)        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)


Remove Infected Connections

Consider this parameter if you simply don't trust your application and/or are experiencing strange connection issues. When your Java application closes a connection, WebLogic Server will actually close it forcibly instead of releasing it back in the pool.

The Oracle documentation states:
Specifies whether a connection will be removed from the connection pool after the application uses the underlying vendor connection object.
If you disable removing infected connections, you must make sure that the database connection is suitable for reuse by other applications.
When set to true (the default), the physical connection is not returned to the connection pool after the application closes the logical connection. Instead, the physical connection is closed and recreated.
When set to false, when the application closes the logical connection, the physical connection is returned to the connection pool and can be reused by the application or by another application.
When this parameter is triggered, the following will appear in the logs:
java.sql.SQLException: Connection closed        at weblogic.jdbc.wrapper.JTAConnection.getXAConn(JTAConnection.java:213)        at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:84)        at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:74)        at weblogic.jdbc.wrapper.Connection.preInvocationHandler(Connection.java:100)


Applicable Versions:
  • Oracle WebLogic Server 11g/12c


Wednesday, July 1, 2015

OSB 11g deployment failing with unexpected error in com.bea.wli.config.deployment.server.ServerRestartRequiredException

Problem:

Are you getting the following error when trying to deploy an OSB project through Ant?

[java] Unexpected error: com.bea.wli.config.deployment.server.ServerRestartRequiredException
[java] No stack trace available.
[java] Problem invoking WLST - Traceback (innermost last):
[java]   File "/home/oracle/deploy/osb/import_deploy.py", line 257, in ?
[java]   File "/home/oracle/deploy/osb/import_deploy.py", line 178, in importToALSBDomain
[java] com.bea.wli.config.deployment.server.ServerRestartRequiredException: Server(s) require restart before session can be activated.
[java]     at com.bea.wli.config.deployment.server.ServerDeploymentReceiver$1.run(ServerDeploymentReceiver.java:225)
[java]     at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
[java]     at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
[java]     at com.bea.wli.config.deployment.server.ServerDeploymentReceiver.prepare(ServerDeploymentReceiver.java:213)
[java]     at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
[java]     at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
[java]     at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
[java]     at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
[java]     at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
[java]     at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Solution:

1. Restart the OSB managed servers and try again.


Applicable Versions:
  • Oracle Service Bus (OSB) 11g (10.3.6+)



Sunday, June 28, 2015

JDeveloper 12c and Updating to OSX 10.10 "Yosemite"

Problem
Recently I finally decided to make the jump from OS X 10.9 to 10.10. It's been a while since it was officially released, but I was trying to be careful about upgrading. I am using my MacBook Pro for daily development work with various softwares, so you can never too careful. I had done my due diligence with most, but I forgot one, JDeveloper, for the simple fact that I haven't been using it all that much lately. According to the compatibility matrix on Oracle's website 12.1.3 is not officially supported on 10.10, so had I taken the time to check I would have known. Thankfully I knew there was nothing in the implementations changes of OS X that should affect compatibility.

When I attempted to start up the program I expected zero issues. However, JDeveloper upon starting up, the loading screen progress bar froze, and then it crashed. No messages, or extra windows to give me an idea of the issue.

JDeveloper Splash Screen
I decided to attempt starting it from the command line in case that might provide some sort of insight by running: $ORACLE_HOME/jdeveloper/JDeveloper.app/Contents/MacOS/JDeveloper. Upon running this I was presented with the following error:


Unfortunately this error in itself is a red herring. So after some additional tinkering with verbosity and debug there was nothing of real use.

Solution
Thankfully I was able to find something on the Oracle Forum, which provides a very simple solution to what could have been a detrimental problem. There seems to be an issue with how the caching works after upgrading, so all that is required is to remove a caching folder using the following command: rm -r ~/.jdeveloper/system12.1.3.0.41.140521.1008/system_cache

That's it. Try starting it up again, and there should be no issue.

Wednesday, June 10, 2015

Sysman link error when installing Oracle Database 11g (11.2.0.4) on Amazon Web Services EC2 Red Hat

Problem:

We downloaded and installed the following Oracle Database 11g (11.2.0.4) 64-bit on a newly created Amazon Web Services EC2 instances (Red Hat Enterprise Linux Server release 7.1).

These are the installation files:

  • p13390677_112040_Linux-x86-64_1of7.zip
  • p13390677_112040_Linux-x86-64_2of7.zip
  • p13390677_112040_Linux-x86-64_3of7.zip
  • p13390677_112040_Linux-x86-64_4of7.zip
  • p13390677_112040_Linux-x86-64_5of7.zip
  • p13390677_112040_Linux-x86-64_6of7.zip
  • p13390677_112040_Linux-x86-64_7of7.zip

Upon installation, the OUI gave an error relating to sysman during the linking process.

The error in the OraInventory (/u01/app/oraInventory/logs/installActions2015-06-10_09-52-46AM.log) log file was:
INFO: /usr/bin/ld: warning: -z lazyload ignored.
/usr/bin/ld: warning: -z nolazyload ignored.
/usr/bin/ld: /u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib//libnmectl.a(nmectlt.o): undefined reference to symbol 'B_DestroyKeyObject'
/usr/bin/ld: note: 'B_DestroyKeyObject' is defined in DSO /u01/app/oracle/product/11.2.0/dbhome_1/lib/libnnz11.so so try adding it to the linker command line
/u01/app/oracle/product/11.2.0/dbhome_1/lib/libnnz11.so: could not read symbols: Invalid operation

INFO: collect2: error: ld returned 1 exit status

INFO: make[1]: *** [/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/emdctl] Error 1

INFO: make[1]: Leaving directory `/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib'

INFO: make: *** [emdctl] Error 2

INFO: End output from spawned process.
INFO: ----------------------------------
INFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'agent nmhs' of makefile '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk'. See '/u01/app/oraInventory/logs/installActions2015-06-10_09-52-46AM.log' for details.
Exception Severity: 1

Solution:

1. Edit this file: /u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk

2. Change the following:
OLD:  $(MK_EMAGENT_NMECTL)
NEW: $(MK_EMAGENT_NMECTL) -lnnz11
3. Retry.

Applicable Versions:
  • Oracle Database 11g (11.2.0.4) 64-bit
  • Amazon EC2
  • Red Hat Enterprise Linux Server release 7.1


References:


 

Wednesday, May 27, 2015

Erik Wahl Keynote - Collaborate 2015 in Las Vegas

Well known graffiti artist and business strategist Erik Wahl gave an outstanding performance during the opening keynote for Collaborate 2015 in Las Vegas. Having a creative mind, he encourages his audience to “break free from business as usual.” The talented artist provided an interactive performance where he spoke about the failure that led him to pursue his passion, and why creativity must be implemented in every aspect of what we do in order to succeed.

He began the keynote with a spectacular performance to U2’s “Beautiful Day”. As Wahl painted, we began to see a face, then glasses, and soon after we made the connection that it was a portrait of Bono. While the song continued and video played at the back of the stage, he continued to put the finishing touches on the painting. Turning to the crowd he began by asking, by a show of hands, how many people in the room could draw. As you’d can expect, in a room full of technical conference attendees, very few raised their hands. He explained why most adults do not see themselves as creative, and why it is necessary to think outside the box when it comes to strategy. He spoke about how all preschoolers can draw with confidence. They want to show you, they have passion, and they have pride in their work. As he said they are “dying to show you their creative energy.” But passion dries up at a young age, as we start favoring analytical and professional interests.

  “All children are artists. The problem is how to remain an artist once he grows up.”

Next Erik spoke about fear. He selected a member of the audience to play Fear Factor. He gave the volunteer an envelope. Wahl gave him two options: perform the instructions or pass the envelope on to one other person to perform the instructions. He chose to accept the challenge. When the volunteer opened the envelope the instructions read that the Bono painting was his to keep. Most people would have passed the challenge on, but as he demonstrated, sometimes risks are worth the reward. Wahl described fear as the acronym False Evidence Appearing Real. Fear is what cripples performance and hinders future progress. Wahl encouraged the audience to take a step away from our comfort zone and look at the future as a blank canvas. It is not always necessary to look at things in an analytical or practiced way. What really matters are the things that cannot be measured or justified. Relationships, passion, and ambition are examples of the driving forces for innovation and creativity of future organizations.

"If you are not afraid, you are not going strong enough." -Erik Wahl

During his second painting, Wahl created a portrait of Steve Jobs. He mentioned one of Jobs’ famous quotes, “Stay hungry, stay foolish.” He admitted to us, that as a child he was told he was messy and didn’t pay attention to detail. Listening to the critics, he gave up drawing and creativity for more and more “adult” activities. He began to memorize and regurgitate rehearsed responses in hopes of fitting into the standard model of excellence. He explained that as we approach adulthood, we are all taught to be increasingly analytical. While these skills are still necessary, they may no longer be sufficient in the competitive and inventive world we live in. After the disappointment of a failed business, Wahl was driven to create art at 30 years old. With his world turned upside down, he had lost what had defined him. He had, like many of us, tied his direct worth to his net worth, and once one of them was gone so was the other. He was encouraged to step away, go travel, and do some soul-searching. To him this meant to begin painting. He immediately felt the creativity unlocking the right side of his brain again. Additionally, he started to understand the best practices of business that he had struggled to understand before. Engaging both right and left sides allows the brain to function fully, using all its ability. Creativity is a practiced and disciplined skill that is extremely useful when being involved with teams, users groups, and other communities. Erik stated that many people say, “I don’t have a creative bone in my body.” They feel as though you are a creative person or you are analytical, but never both. Wahl explains that this is not the case. While society encourages individuals to become increasingly analytical, Wahl is encouraging people to use the right side of their brain and explore their creativity. Deep down we all have things that ignite passion and creativity in us; we have inspiration all around us. Our world is becoming increasingly competitive, with new ideas emerging which break outside of the comfort zone. Being innovative drives customer loyalty; with creativity comes unconventional solutions; with failure comes an opportunity to grow. Wahl motivated the audience to be confident, to be courageous, and to face the fear of failure.  He concluded with a final portrait of one of the most creative minds that ever lived, Albert Einstein. In his inspiring performance he reached out to each and every one of us to change the way we approach business, and to expand out of our comfort zone. With his energetic style and beautiful works of art, he captivated and motivated the audience to live life a little more creatively.




Tuesday, May 26, 2015

The low-cost approach to monitoring Oracle products and environments

There are many capable monitoring tools out there today on the market that provide true end-to-end monitoring and management of your Oracle infrastructure. The first product that comes to mind is Oracle Enterprise Manager 12c Cloud Control. It is definitely a must have for any Oracle shop and we highly recommend it.

Unfortunately, for the small or midsized company, the total cost of ownership of OEM may be prohibitive.

At Raastech, we've implemented a series of scripts at numerous customers that perform a reasonably adequate job in monitoring and managing the more important areas of Oracle WebLogic Server, Oracle SOA Suite, and the Oracle Database.
Monitoring 
Infrastructure monitoring and performance reports. 
Alerting 
Email alerts upon trigger of critical events occur or if predefined thresholds are exceeded. 
Management 
Full and incremental backups of middleware and database, log rotation, archiving, and purging.
Take a look at some of the emails that we send out. If you're looking for an extremely low-cost and basic solution for your Oracle environments, reach out to us and we can help.


Oracle WebLogic Server Status Report



Weekly Oracle Database Report



Oracle SOA Suite Component Instance Performance Report



RMAN Backup Report




 

Tuesday, May 12, 2015

BEA-382515: Callout to Java resulted in null exception

Problem:

Getting the following error testing an OSB 11g service in the OSB Console:
BEA-382515: Callout to java method "public static java.lang.String com.amway.common.XMLJSONConverter.convertXML2JSON(java.lang.String) throws java.lang.Exception" resulted in exception: null
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at stages.transform.runtime.JavaCalloutRuntimeStep$1.run(JavaCalloutRuntimeStep.java:183)

Analysis:

The osb_server1.out file also reports this error:
<May 12, 2015 4:55:32 PM EDT> <Error> <OSB Transform> <BEA-382515> <Callout to java method "public static java.lang.String com.amway.common.XMLJSONConverter.convertXML2JSON(java.lang.String) throws java.lang.Exception" resulted in exception: null
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at stages.transform.runtime.JavaCalloutRuntimeStep$1.run(JavaCalloutRuntimeStep.java:183)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.NoClassDefFoundError: nu/xom/Serializer
        at com.amway.common.XMLJSONConverter.convertXML2JSON(XMLJSONConverter.java:42)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        Truncated. see log file for complete stacktrace
>
As you can see, the class is not found for nu/xom/Serializer, which is a part of the xom.jar file.


Solution:

1. Copy xom-1.2.6.jar to the following folder:
$DOMAIN_HOME/soa_domain/lib/xom-1.2.6.jar
2. Bounce both osb_server1


Applicable Versions:
  • Oracle WebLogic Server 11g
  • Oracle Service Bus (OSB) 11g 

 

Monday, May 4, 2015

Version Uncontrolled @ COLLABORATE15

With another COLLABORATE complete and I have been back in the office I have taken the time to commit a lot to memory and added many tasks to research further into topics related to many of the presentations I attended. Over all I would have to say the the conference was successful from my perspective. I put more focus on the database, since even though I have skills in developing PL/SQL and doing simple database setups, my knowledge in database operations is not where I would like it. One of the key things to realize is that in most cases applications are using databases. Products under the Fusion Middleware label are no different. Moving forward I am looking to expand more into database performance as a way to improve Middleware performance focused around SOA Suite and OSB. While I was learning I also had a chance to share some of my experiences, and meet many people.

I would like to thank everyone who came to my presentation. "Version Uncontrolled ..." took some time to focus on a few version control technologies and concepts, and followed with workflows. The various workflows can be found on our company SlideShare or on the Raastech Library. In the future, I hope to be given the privilege of presenting again. I would like to thank all of the IOUG Directors for their time and effort in on this COLLABORATE as well as all the other efforts we don't see going on.


Friday, May 1, 2015

BEA-149205 Failed to initialize the application "Service Bus Framework Starter Application" due to OSB system user authentication failed

Problem:

Getting the following error when starting up the AdminServer on Oracle WebLogic Server 12c with Oracle Service Bus 12c installed:
####<Apr 20, 2015 11:44:21 PM EDT> <Error> <Deployer> <soahost1> <AdminServer> <[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <236cd04a-cdd4-4c4c-bf1b-f4ca7b2fe3fe-00000004> <1429587861652> <BEA-149205> <Failed to initialize the application "Service Bus Framework Starter Application" due to error java.lang.RuntimeException: OSB system user authentication failed
java.lang.RuntimeException: OSB system user authentication failed
        at com.bea.wli.sb.security.SecurityModuleImpl.runAlsbPrivilegedAction(SecurityModuleImpl.java:345)
        at com.bea.wli.sb.security.SecurityModuleImpl.jndiBind(SecurityModuleImpl.java:415)
        at com.bea.alsb.platform.weblogic.WlsJNDIService.bind(WlsJNDIService.java:46)
        at com.bea.wli.sb.init.FrameworkStarter$ALSBClusterTimerJndiAccess.bindClusterTimer(FrameworkStarter.java:362)
        at com.bea.wli.timer.ClusterTimerService.initialize(ClusterTimerService.java:87)
        at com.bea.wli.sb.init.FrameworkStarter._preStart(FrameworkStarter.java:215)
        at com.bea.wli.sb.init.FrameworkStarter.access$000(FrameworkStarter.java:75)
        at com.bea.wli.sb.init.FrameworkStarter$1.run(FrameworkStarter.java:94)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
        at weblogic.security.Security.runAs(Security.java:61)


Solution:

1. Login to the WebLogic Server Administration Console

2. Navigate to Security Realms > myrealm > Users and Groups.

3. Click on "New" to create a new user using the following:
NAME: alsb-system-user 
PASSWORD: (any password) 
DESCRIPTION: The ALSB system user is a built-in system account which belongs to the ALSBSystem role. As such it has access to ALSBs internal artifacts. The password for this account is automatically changed when the admin server boots to prevent direct access to this account.
4. Save and restart the AdminServer.



Applicable Versions:
  • Oracle WebLogic Server 12c (12.1.3)
  • Oracle Service Bus (OSB) 12c 

 

Wednesday, April 22, 2015

The WebLogic Server 12c .PID file

One of the managed servers (wls_soa1) on our Oracle WebLogic Server 12c installation was showing a state of STARTING, but there was no process running on the Linux server:
 

Under the $DOMAIN_HOME/servers/wls_soa1/data/nodemanager folder, you will find multiple files:
oracle@soahost1:/u01/share/oracle/middleware/config/domains/soa_domain/servers/wls_soa1/data/nodemanager> ls -l
total 40 
-rw-r----- 1 oracle oinstall 174 Apr 14 22:14 boot.properties 
-rw-r----- 1 oracle oinstall 427 Apr 14 22:14 startup.properties 
-rw-r--r-- 1 oracle oinstall   6 Apr 22 14:41 wls_soa1.pid 
-rw-r----- 1 oracle oinstall  13 Apr 22 14:43 wls_soa1.state 
-rw-r----- 1 oracle oinstall  31 Mar 29 12:07 wls_soa1.url

The content of the .pid file will show RUNNING:
oracle@soahost1:/u01/share/oracle/middleware/config/domains/soa_domain/servers/wls_soa1/data/nodemanager> cat wls_soa1.state 
STARTING:N:N

Delete this .pid file:
oracle@soahost1:/u01/share/oracle/middleware/config/domains/soa_domain/servers/wls_soa1/data/nodemanager> rm wls_soa1.state

When you refresh the page on the WebLogic Admin Console, it will default to SHUTDOWN now:

Only after the console refresh will a new .pid file be created with the SHUTDOWN state:
oracle@soahost1:/u01/share/oracle/middleware/config/domains/soa_domain/servers/wls_soa1/data/nodemanager> cat wls_soa1.state 
SHUTDOWN:N:N



Applicable Versions:
  • Oracle WebLogic Server 12c (12.1.3) 

 

Friday, April 10, 2015

Incredibly Useful SQL Queries for Oracle SOA Suite 11g

These are the collection of the incredibly useful SQL queries for Oracle SOA Suite 11g. Parameters highlighted are customizable.

All SQL should should be executed as the <PREFIX>_SOAINFRA user.



Component avg/min/max performance

SELECT 'Mediator' type,
       SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       DECODE(component_state, '0', 'Completed', '16', 'Stale', '2', 'Faulted', '4', 'Recovery Needed', '8', 'Running') state,
       TO_CHAR(AVG((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),18,4))),'999990.000') Avg,
       TO_CHAR(MIN((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),18,4))),'999990.000') Min,
       TO_CHAR(MAX((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),18,4))),'999990.000') Max,
       COUNT(1) count
FROM   mediator_instance
WHERE  created_time >= TO_DATE('2015-04-22 00:00', 'YYYY-MM-DD HH24:MI')
--AND    created_time <= TO_DATE('2013-07-03 23:59', 'YYYY-MM-DD HH24:MI')
GROUP BY component_name, component_state
UNION
SELECT 'BPEL' type,
       domain_name partition,
       component_name component,
       DECODE(state,'5','Completed','9','Stale','10','Faulted') State,
       TO_CHAR(AVG((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') Avg,
       TO_CHAR(MIN((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') Min,
       TO_CHAR(MAX((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') Max,
       COUNT(1) count
FROM   cube_instance
WHERE  creation_date >= TO_DATE('2015-04-22 00:00', 'YYYY-MM-DD HH24:MI')
--AND    creation_date <= TO_DATE('2013-07-03 23:59', 'YYYY-MM-DD HH24:MI')
GROUP BY domain_name, component_name, state
ORDER BY 2, 3















Mediator avg/min/max performance

SELECT SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       source_action_name action,
       DECODE(component_state, '0', 'Completed', '16', 'Stale', '2', 'Faulted', '4', 'Recovery Needed', '8', 'Running') state,
       TO_CHAR(AVG((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),18,4))),'999990.000') Avg,
       TO_CHAR(MIN((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),18,4))),'999990.000') Min,
       TO_CHAR(MAX((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),18,4))),'999990.000') Max,
       COUNT(1) count
FROM   mediator_instance
-- Last three hours
WHERE  created_time >= sysdate - 0.125
--AND    COMPONENT_NAME LIKE '%%'
GROUP BY component_name, source_action_name, component_state
ORDER BY component_name, source_action_name, component_state












BPEL avg/min/max performance

SELECT domain_name,
       component_name,
       DECODE(state,'5','Complete','9','Stale','10','Faulted') State,
       TO_CHAR(AVG((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') Avg,
       TO_CHAR(MIN((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') Min,
       TO_CHAR(MAX((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') Max,
       COUNT(1) count
FROM   cube_instance
-- Last 3 hours
WHERE  CREATION_DATE >= sysdate - 0.125
--AND    COMPONENT_NAME LIKE '%%'
--AND    COMPOSITE_NAME LIKE '%%'
GROUP BY domain_name, component_name, state
ORDER BY component_name, state



Composite state count

SELECT SUBSTR(composite_dn, 1, INSTR(composite_dn,'/')-1) partition,
       SUBSTR(composite_dn, INSTR(composite_dn,'/')+1, INSTR(composite_dn,'!')-INSTR(composite_dn,'/')-1) component,
       DECODE(state,'1','Complete','0','Running','3','Faulted','32','Unknown','5','Recovery Needed','7','Recovery Needed','64','Stale') State,
       count(1) Count
FROM   composite_instance 
WHERE  composite_dn LIKE '%%'
GROUP BY SUBSTR(composite_dn, 1, INSTR(composite_dn,'/')-1),SUBSTR(composite_dn, INSTR(composite_dn,'/')+1, INSTR(composite_dn,'!')-INSTR(composite_dn,'/')-1),state
ORDER BY SUBSTR(composite_dn, 1, INSTR(composite_dn,'/')-1),SUBSTR(composite_dn, INSTR(composite_dn,'/')+1, INSTR(composite_dn,'!')-INSTR(composite_dn,'/')-1),state



Composite instances and state

SELECT id Instance,
       SUBSTR(composite_dn, 1, INSTR(composite_dn,'/')-1) partition,
       SUBSTR(composite_dn, INSTR(composite_dn,'/')+1, INSTR(composite_dn,'!')-INSTR(composite_dn,'/')-1) component,
       DECODE(state,'1','Complete','0','Running','3','Faulted','32','Unknown','5','Recovery Needed','7','Recovery Needed','64','Stale') State
FROM   composite_instance 
WHERE  composite_dn LIKE '%%'
--AND    state = 0 -- 0 = Running 1 = Completed 3 = Faulted 5&7 = Recovery Needed 32 = Unknown 64 = Stale
ORDER BY SUBSTR(composite_dn, 1, INSTR(composite_dn,'/')-1),SUBSTR(composite_dn, INSTR(composite_dn,'/')+1, INSTR(composite_dn,'!')-INSTR(composite_dn,'/')-1), state



Mediator state counts

SELECT n.partition,
       n.component,
       nvl(c.completed,0) Completed,
       nvl(f.failed,0) Failed,
       nvl(rn.Recovery_needed,0) Recovery_Needed,
       nvl(r.running,0) Running,
       nvl(s.stale,0) Stale,
       nvl(t.total,0) Total
FROM
(SELECT Unique SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component
 FROM mediator_instance) n,
(Select SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       count(1) Completed
       from mediator_instance
       WHERE component_state = 0
       AND created_time >= sysdate - 0.125
       GROUP BY SUBSTR(component_name, 1, INSTR(component_name,'/')-1), SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1)) c ,
(Select SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       count(1) Failed
       from mediator_instance
       WHERE component_state = 2
       AND created_time >= sysdate - 0.125
       GROUP BY SUBSTR(component_name, 1, INSTR(component_name,'/')-1), SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1)) f ,
(Select SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       count(1) Recovery_Needed
       from mediator_instance
       WHERE component_state = 4
       AND created_time >= sysdate - 0.125
       GROUP BY SUBSTR(component_name, 1, INSTR(component_name,'/')-1), SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1)) rn ,
(Select SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       count(1) Running
       from mediator_instance
       WHERE component_state = 8
       AND created_time >= sysdate - 0.125
       GROUP BY SUBSTR(component_name, 1, INSTR(component_name,'/')-1), SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1)) r ,
(Select SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       count(1) Stale
       from mediator_instance
       WHERE component_state = 16
       AND created_time >= sysdate - 0.125
       GROUP BY SUBSTR(component_name, 1, INSTR(component_name,'/')-1), SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1)) s ,
(Select SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       count(1) Total
       from mediator_instance
       WHERE created_time >= sysdate - 0.125
       GROUP BY SUBSTR(component_name, 1, INSTR(component_name,'/')-1), SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1)) t
where n.partition = c.partition(+)
AND   n.component = c.component(+)
AND   n.partition = f.partition(+)
AND   n.component = f.component(+)
AND   n.partition = rn.partition(+)
AND   n.component = rn.component(+)
AND   n.partition = r.partition(+)
AND   n.component = r.component(+)
AND   n.partition = s.partition(+)
AND   n.component = s.component(+)
AND   n.partition = t.partition(+)
AND   n.component = t.component(+)
Order by --failed desc,
         --completed desc,
         partition,
         component;



Faulted Mediator instances

SELECT composite_instance_id Instance_ID,
       SUBSTR(composite_dn, 1, INSTR(composite_dn,'/')-1) partition,
       SUBSTR(composite_dn, INSTR(composite_dn,'/')+1, INSTR(composite_dn,'!')-INSTR(composite_dn,'/')-1) component,
       error_message
FROM COMPOSITE_INSTANCE_FAULT where composite_instance_id in
(SELECT composite_instance_id 
 FROM mediator_instance 
 WHERE component_state = 2
 AND created_time >= sysdate - 0.125)



Faulted BPEL instances

 SELECT composite_instance_id Instance_ID,
       SUBSTR(composite_dn, 1, INSTR(composite_dn,'/')-1) partition,
       SUBSTR(composite_dn, INSTR(composite_dn,'/')+1, INSTR(composite_dn,'!')-INSTR(composite_dn,'/')-1) component,
       error_message
FROM COMPOSITE_INSTANCE_FAULT where composite_instance_id in
(SELECT cikey
 FROM cube_instance
 WHERE state = 10
 AND   creation_date >= sysdate - 0.125)



Mediator end-to-end Flows that took over 60 seconds

SELECT SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       source_action_name action,
       DECODE(component_state, '0', 'Completed', '16', 'Stale', '2', 'Faulted', '4', 'Recovery Needed', '8', 'Running') state,
       COUNT(1) count,
       '60' seconds
FROM   mediator_instance
WHERE  TO_CHAR(composite_creation_date, 'YYYY-MM-DD HH24:MI') >= TO_CHAR(sysdate-0.125, 'YYYY-MM-DD HH24:MI')
AND    TO_CHAR(composite_creation_date, 'YYYY-MM-DD HH24:MI') <= TO_CHAR(sysdate, 'YYYY-MM-DD HH24:MI')
AND    parent_id IS NULL
AND    TO_NUMBER(TO_CHAR((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-composite_creation_date),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-composite_creation_date),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-composite_creation_date),18,4)),'999990.0')) > 60
GROUP BY component_name, source_action_name, component_state
ORDER BY component_name, source_action_name



Duration of time for Mediator reference calls

SELECT composite_instance_id,
       TO_CHAR(composite_creation_date, 'YYYY-MM-DD HH24:MI') creation_date,
       -- SUBSTR(component_name, 1, INSTR(component_name,'/')-1) partition,
       SUBSTR(component_name, INSTR(component_name,'/')+1, INSTR(component_name,'!')-INSTR(component_name,'/')-1) component,
       -- source_action_name action,
       DECODE(component_state, '0', 'Completed', '16', 'Stale', '2', 'Faulted', '4', 'Recovery Needed', '8', 'Running') state,
       TO_CHAR((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-created_time),18,4)),'999990.000') component_duration,
       TO_CHAR((TO_NUMBER(SUBSTR(TO_CHAR(updated_time-composite_creation_date),12,2))*60*60) + (TO_NUMBER(SUBSTR(TO_CHAR(updated_time-composite_creation_date),15,2))*60) + TO_NUMBER(SUBSTR(TO_CHAR(updated_time-composite_creation_date),18,4)),'999990.000') composite_duration
FROM   mediator_instance
WHERE  TO_CHAR(composite_creation_date, 'YYYY-MM-DD HH24:MI') >= '2015-04-22 09:00'
--AND    parent_id IS NULL
AND    component_name LIKE '%%'
AND    source_action_name = '%%'
ORDER BY creation_date, component_name, source_action_name, component_state





Applicable Versions:
  • Oracle SOA Suite 11g