Wednesday, December 21, 2016

OAM Coherence General SSLEngine problem - Certificates does not conform to algorithm constraints

Oracle Access Management 11.1.2.3.0 running on JDK 1.7.0.121 have the following warning spawn on the logs every a couple of seconds:

<Dec 19, 2016 5:08:07 PM EET> <Warning> <Coherence> <BEA-000000> <2016-12-19 17:08:07.764/977.031 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListener1, member=1): TcpDatagramSocket{bind=ServerSocket[addr=/x.x.x.x,localport=9095]}, exception regarding peer oam.xxx.com/x.x.x.x:9095, General SSLEngine problem; Certificates does not conform to algorithm constraints>
<Dec 19, 2016 5:08:07 PM EET> <Warning> <Coherence> <BEA-000000> <2016-12-19 17:08:07.764/977.031 Oracle Coherence GE 3.7.1.13 <Warning> (thread=PacketListener1, member=1): TcpDatagramSocket{bind=ServerSocket[addr=/x.x.x.x,localport=9095]}, exception regarding peer /x.x.x.x:48698, Received fatal alert: certificate_unknown>

This is due to The SSL MD5withRSA support is deprecated and RSA keySize < 1024 is disabled by default as of JDK 1.7 release 95.

A quick and temporal (unsecure) fix is to edit java.security on jdk/jre/lib/security/java.security
and remove MD5withRSA from
jdk.tls.disabledAlgorithms=SSLv3, MD5withRSA, DH keySize < 768

and also remove MD5 and lower the RSA keySize to 512 in
jdk.certpath.disabledAlgorithms=MD2, MD5, RSA keySize < 1024


To permanently solve this error certificate need to be replaced with a stronger one.

First we need to recover keystore password using em.

There is a method called getPortableCredential in Application Defined MBeans->com.oracle.jps->Domain: <your domain name>->JpsCredentialStore JpsCredentialStore found with System MBean Browser.

Execute using OAM_STORE for parameter 1 and coh for parameter 2.

Inside domain_home/config/fmwconfig we create a new certificate:
keytool -genkey -alias admin -keyalg RSA -keysize 2048 -dname "CN=\"administrator ou=oam\", o=Oracle, C=US" -validity 3650 -keypass 9tgsga3ohf8let75019jfk2tga -keystore .cohstore.jks.new -storetype jks -storepass 9tgsga3ohf8let75019jfk2tga

Then we export the certificate:
keytool -export -alias admin -file cohadmin.cert -keystore .cohstore.jks.new -storepass 9tgsga3ohf8let75019jfk2tga -storetype jks
Certificate stored in file <cohadmin.cert>

And then we import the certificate back as assertion-cert:

keytool -importcert -alias assertion-cert -trustcacerts -file cohadmin.cert -keystore .cohstore.jks.new -storetype jks -storepass 9tgsga3ohf8let75019jfk2tga
Certificate already exists in keystore under alias <admin>
Do you still want to add it? [no]:  yes
Certificate was added to keystore

Finally we stop all servers and replace cohstore.jks with cohstore.jks.new

Starting all sevrers and error is gone away.

Reference doc: 1986560.1




3 comments:

  1. Hi,

    I tried what you suggested (and as the Oracle support article suggested) and I think that might fix the original SSL problem, but when I start the Adminserver with the new .cohstore.jks, it is encountering error:

    <2017-05-29 11:43:36.173/104.911 Oracle Coherence GE 3.7.1.13 (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error while starting cluster: java.lang.IllegalArgumentException: Invalid configuration:


    file:/apps/oracle/Middleware/user_projects/domains/IDMDomain/config/fmwconfig/.cohstore.jks




    SunX509


    PeerX509

    file:/apps/oracle/Middleware/user_projects/domains/IDMDomain/config/fmwconfig/.cohstore.jks
    26slajavn0vg2l8d747hnnd7og




    at com.tangosol.net.ssl.SSLSocketProvider.setConfig(SSLSocketProvider.java:437)
    at com.tangosol.net.SocketProviderFactory.createProvider(SocketProviderFactory.java:77)
    at com.tangosol.net.SocketProviderFactory.ensureProvider(SocketProviderFactory.java:152)
    at com.tangosol.coherence.component.net.Cluster.configureSockets(Cluster.CDB:28)
    at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:28)
    at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)

    It seems like something does not like the new .cohstore.jks?

    FYI, I tried "keytool -list" with the new .cohstore.jks and that works fine so not sure what is causing the Adminserver problem?

    ReplyDelete
  2. I figured out my problem... I didn't notice that the first keytool command had the password in two different parameters, so I had left one of them set to the example value above.

    ReplyDelete