Getting "httpd.worker': double free or corruption" when starting up OHS

Problem

Starting up OHS fails.

This error appears in ${OHS_LOGS}/ohs1.log:

--------
17/06/17 02:37:46 Start process
--------
/u01/app/oracle/middleware/Oracle_WT1/ohs/bin/apachectl startssl: execing httpd
[Sat Jun 17 02:37:46 2017] [warn] Errors will be logged into /u01/app/oracle/middleware/Oracle_WT1/instances/obiee/diagnostics/logs/OHS/ohs1/ohs1.log
[Sat Jun 17 02:37:47 2017] [warn] Errors will be logged into /u01/app/oracle/middleware/Oracle_WT1/instances/obiee/diagnostics/logs/OHS/ohs1/ohs1.log
*** Error in `/u01/app/oracle/middleware/Oracle_WT1/ohs/bin/httpd.worker': double free or corruption (out): 0x0000000001821300 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x7f3a1f9ab503]
/u01/app/oracle/middleware/Oracle_WT1/lib/libnnz11.so(nzumfree+0x64)[0x7f3a1f03aefa]
/u01/app/oracle/middleware/Oracle_WT1/lib/libnnz11.so(nztiFIC_Free_Identity_Contents+0x45)[0x7f3a1f06f2d3]
/u01/app/oracle/middleware/Oracle_WT1/lib/libnnz11.so(nztiDI_Destroy_Identity+0x3a)[0x7f3a1f06f268]
/u01/app/oracle/middleware/Oracle_WT1/lib/libnnz11.so(nztiFIL_Free_Identity_List+0x5a)[0x7f3a1f06f21a]
/u01/app/oracle/middleware/Oracle_WT1/lib/libnnz11.so(nztnFPC_Free_Persona_Contents+0x45)[0x7f3a1f072ec1]

If you comment out the ssl.conf line in httpd.conf, it starts up fine.


Solution

1. This was because the certificate chain was not complete in trust.jks.

For example, the identity cert was signed by a CA4, so the trust was missing the intermediate and root certs for that CA.

To import those certificates into the trust keystore:

keytool -import -alias PRODCA4 -file PRODCA4.crt -keystore /u01/app/oracle/middleware/keystore/trust.jks

keytool -import -alias PRODTreasury -file PRODTreasury.crt -keystore /u01/app/oracle/middleware/keystore/trust.jks 

Applicable Versions

• Oracle HTTP Server (OHS) 11g (11.1.1.9.0)