Unable to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSource'

Question

I am getting below error in prod mode for (Java8+Oauth2+MySql+Hazelcast+no clustered http sessions) combination. Dev mode has worked fine.

Unable to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSou
rce'; nested exception is javax.management.InstanceAlreadyExistsException: com.z
axxer.hikari:name=dataSource,type=HikariDataSource

In the log, first part of the datasource (hazelcast) creation says [dev] mode. Not sure this is deliberate.

Kindly help me in resolving the issue.

log

Dec 12, 2014 2:44:11 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive D:\tools\apache-tomcat-7.0.55\webapps\re
tailenergy.war
[INFO] com.fg.re.ApplicationWebXml - Running with Spring profile(s) : prod
2014-12-12 14:44:19.125  WARN 1368 --- [ost-startStop-1] o.s.b.l.LoggingApplicat
ionListener       : Logging environment value '-Djava.util.logging.config.file="
D:\tools\apache-tomcat-7.0.55\conf\logging.properties"' cannot be opened and wil
l be ignored (using default location instead)
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more in
fo.
[DEBUG] com.fg.re.config.AsyncConfiguration - Creating Async Task Executor
[DEBUG] com.fg.re.config.MetricsConfiguration - Registering JVM gauges
[INFO] com.fg.re.config.MetricsConfiguration - Initializing Metrics JMX reportin
g
[INFO] com.hazelcast.instance.DefaultAddressPicker - null [dev] [3.2.5] Prefer I
Pv4 stack is true.
[INFO] com.hazelcast.instance.DefaultAddressPicker - null [dev] [3.2.5] Picked A
ddress[192.168.1.9]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localp
ort=5701], bind any local is true
[INFO] com.hazelcast.system - [192.168.1.9]:5701 [dev] [3.2.5] Hazelcast 3.2.5 (
20140814) starting at Address[192.168.1.9]:5701
[INFO] com.hazelcast.system - [192.168.1.9]:5701 [dev] [3.2.5] Copyright (C) 200
8-2014 Hazelcast.com
[INFO] com.hazelcast.instance.Node - [192.168.1.9]:5701 [dev] [3.2.5] Creating M
ulticastJoiner
[INFO] com.hazelcast.core.LifecycleService - [192.168.1.9]:5701 [dev] [3.2.5] Ad
dress[192.168.1.9]:5701 is STARTING
[INFO] com.hazelcast.cluster.MulticastJoiner - [192.168.1.9]:5701 [dev] [3.2.5]



Members [1] {
        Member [192.168.1.9]:5701 this
}

[INFO] com.hazelcast.core.LifecycleService - [192.168.1.9]:5701 [dev] [3.2.5] Ad
dress[192.168.1.9]:5701 is STARTED
[DEBUG] com.fg.re.config.CacheConfiguration - Starting HazelcastCacheManager
[INFO] com.hazelcast.util.HealthMonitor - [192.168.1.9]:5701 [dev] [3.2.5] memor
y.used=242.6M, memory.free=219.4M, memory.total=462.0M, memory.max=891.0M, memor
y.used/total=52.51%, memory.used/max=27.23%, load.process=-100.00%, load.system=
100.00%, load.systemAverage=-100.00%, thread.count=38, thread.peakCount=38, even
t.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.operat
ion.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.
size=0, executor.q.system.size=0, executor.q.operation.size=0, executor.q.priori
tyOperation.size=0, executor.q.response.size=0, operations.remote.size=0, operat
ions.running.size=0, proxy.count=0, clientEndpoint.count=0, connection.active.co
unt=0, connection.count=0
[DEBUG] com.fg.re.config.DatabaseConfiguration - Configuring Datasource
[DEBUG] com.zaxxer.hikari.HikariConfig - HikariCP pool HikariPool-0 configuratio
n:
[DEBUG] com.zaxxer.hikari.HikariConfig - autoCommit......................true
[DEBUG] com.zaxxer.hikari.HikariConfig - catalog.........................
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionCustomizer............com.zax
xer.hikari.HikariConfig$1@5a3cdbda
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionCustomizerClassName...
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionInitSql...............
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionTestQuery.............
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionTimeout...............30000
[DEBUG] com.zaxxer.hikari.HikariConfig - dataSource......................
[DEBUG] com.zaxxer.hikari.HikariConfig - dataSourceClassName.............com.mys
ql.jdbc.jdbc2.optional.MysqlDataSource
[DEBUG] com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................
[DEBUG] com.zaxxer.hikari.HikariConfig - dataSourceProperties............{user=r
oot, url=jdbc:mysql://localhost:3306/retailenergy, password=<masked>, prepStmtCa
cheSqlLimit=2048, cachePrepStmts=true, useServerPrepStmts=true, prepStmtCacheSiz
e=250}
[DEBUG] com.zaxxer.hikari.HikariConfig - driverClassName.................
[DEBUG] com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000
[DEBUG] com.zaxxer.hikari.HikariConfig - initializationFailFast..........true
[DEBUG] com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
[DEBUG] com.zaxxer.hikari.HikariConfig - jdbc4ConnectionTest.............false
[DEBUG] com.zaxxer.hikari.HikariConfig - jdbcUrl.........................
[DEBUG] com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........0
[DEBUG] com.zaxxer.hikari.HikariConfig - maxLifetime.....................1800000

[DEBUG] com.zaxxer.hikari.HikariConfig - maximumPoolSize.................10
[DEBUG] com.zaxxer.hikari.HikariConfig - metricRegistry..................com.cod
ahale.metrics.MetricRegistry@4152b59f
[DEBUG] com.zaxxer.hikari.HikariConfig - minimumIdle.....................10
[DEBUG] com.zaxxer.hikari.HikariConfig - password........................<masked
>
[DEBUG] com.zaxxer.hikari.HikariConfig - poolName........................HikariP
ool-0
[DEBUG] com.zaxxer.hikari.HikariConfig - readOnly........................false
[DEBUG] com.zaxxer.hikari.HikariConfig - registerMbeans..................false
[DEBUG] com.zaxxer.hikari.HikariConfig - threadFactory...................
[DEBUG] com.zaxxer.hikari.HikariConfig - transactionIsolation............
[DEBUG] com.zaxxer.hikari.HikariConfig - username........................
[INFO] com.zaxxer.hikari.HikariDataSource - HikariCP pool HikariPool-0 is starti
ng.
[DEBUG] com.fg.re.config.DatabaseConfiguration - Configuring Liquibase
[INFO] com.fg.re.config.hazelcast.HazelcastCacheRegionFactory - Starting up Haze
lcastCacheRegionFactory
[DEBUG] com.fg.re.config.MailConfiguration - Configuring mail server
[INFO] com.fg.re.config.WebConfigurer - Web application configuration, using pro
files: [prod]
[DEBUG] com.fg.re.config.WebConfigurer - Initializing Metrics registries
[DEBUG] com.fg.re.config.WebConfigurer - Registering Metrics Filter
[DEBUG] com.fg.re.config.WebConfigurer - Registering Metrics Servlet
[DEBUG] com.fg.re.config.WebConfigurer - Registering Caching HTTP Headers Filter

[DEBUG] com.fg.re.config.WebConfigurer - Registering static resources production
 Filter
[DEBUG] com.fg.re.config.WebConfigurer - Registering GZip Filter
[INFO] com.fg.re.config.WebConfigurer - Web application fully configured
[INFO] com.fg.re.Application - Running with Spring profile(s) : [prod]
[INFO] com.fg.re.config.ThymeleafConfiguration - loading non-reloadable mail mes
sages resources
[WARN] org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebAppl
icationContext - Exception encountered during context initialization - cancellin
g refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean wit
h name 'mbeanExporter' defined in class path resource [org/springframework/conte
xt/annotation/MBeanExportConfiguration.class]: Invocation of init method failed;
 nested exception is org.springframework.jmx.export.UnableToRegisterMBeanExcepti
on: Unable to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSou
rce'; nested exception is javax.management.InstanceAlreadyExistsException: com.z
axxer.hikari:name=dataSource,type=HikariDataSource
        at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.initializeBean(AbstractAutowireCapableBeanFactory.java:1554) ~[spring-be
ans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) ~[spring-beans
-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.createBean(AbstractAutowireCapableBeanFactory.java:475) ~[spring-beans-4
.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getOb
ject(AbstractBeanFactory.java:302) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEAS
E]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-4.0.8.RELEA
SE.jar:4.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBe
an(AbstractBeanFactory.java:298) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean
(AbstractBeanFactory.java:193) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.
preInstantiateSingletons(DefaultListableBeanFactory.java:706) ~[spring-beans-4.0
.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finish
BeanFactoryInitialization(AbstractApplicationContext.java:762) ~[spring-context-
4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refres
h(AbstractApplicationContext.java:482) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.
RELEASE]
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationConte
xt.refresh(EmbeddedWebApplicationContext.java:109) [spring-boot-1.1.9.RELEASE.ja
r:1.1.9.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.
java:691) [spring-boot-1.1.9.RELEASE.jar:1.1.9.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java
:320) [spring-boot-1.1.9.RELEASE.jar:1.1.9.RELEASE]
        at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringA
pplicationBuilder.java:142) [spring-boot-1.1.9.RELEASE.jar:1.1.9.RELEASE]
        at org.springframework.boot.context.web.SpringBootServletInitializer.cre
ateRootApplicationContext(SpringBootServletInitializer.java:89) [spring-boot-1.1
.9.RELEASE.jar:1.1.9.RELEASE]
        at org.springframework.boot.context.web.SpringBootServletInitializer.onS
tartup(SpringBootServletInitializer.java:51) [spring-boot-1.1.9.RELEASE.jar:1.1.
9.RELEASE]
        at org.springframework.web.SpringServletContainerInitializer.onStartup(S
pringServletContainerInitializer.java:175) [spring-web-4.0.8.RELEASE.jar:4.0.8.R
ELEASE]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContex
t.java:5479) [catalina.jar:7.0.55]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
[catalina.jar:7.0.55]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase
.java:901) [catalina.jar:7.0.55]
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:87
7) [catalina.jar:7.0.55]
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
 [catalina.jar:7.0.55]
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1083
) [catalina.jar:7.0.55]
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:
1880) [catalina.jar:7.0.55]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51
1) [na:1.8.0_25]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_25
]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142) [na:1.8.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617) [na:1.8.0_25]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
Caused by: org.springframework.jmx.export.UnableToRegisterMBeanException: Unable
 to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSource'; nest
ed exception is javax.management.InstanceAlreadyExistsException: com.zaxxer.hika
ri:name=dataSource,type=HikariDataSource
        at org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstan
ce(MBeanExporter.java:608) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.jmx.export.MBeanExporter.registerBeans(MBeanExpor
ter.java:533) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.jmx.export.MBeanExporter.afterPropertiesSet(MBean
Exporter.java:415) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1613) ~[spring
-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.initializeBean(AbstractAutowireCapableBeanFactory.java:1550) ~[spring-be
ans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        ... 28 common frames omitted
Caused by: javax.management.InstanceAlreadyExistsException: com.zaxxer.hikari:na
me=dataSource,type=HikariDataSource
        at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437) ~[na
:1.8.0_25]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRep
ository(DefaultMBeanServerInterceptor.java:1898) ~[na:1.8.0_25]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamic
MBean(DefaultMBeanServerInterceptor.java:966) ~[na:1.8.0_25]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(
DefaultMBeanServerInterceptor.java:900) ~[na:1.8.0_25]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(D
efaultMBeanServerInterceptor.java:324) ~[na:1.8.0_25]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.j
ava:522) ~[na:1.8.0_25]
        at org.springframework.jmx.support.MBeanRegistrationSupport.doRegister(M
BeanRegistrationSupport.java:195) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEA
SE]
        at org.springframework.jmx.export.MBeanExporter.registerBeanInstance(MBe
anExporter.java:653) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        at org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstan
ce(MBeanExporter.java:598) ~[spring-context-4.0.8.RELEASE.jar:4.0.8.RELEASE]
        ... 32 common frames omitted
[INFO] com.fg.re.config.hazelcast.HazelcastCacheRegionFactory - Shutting down Ha
zelcastCacheRegionFactory
[INFO] com.zaxxer.hikari.pool.HikariPool - HikariCP pool HikariPool-0 is shuttin
g down.
[DEBUG] com.zaxxer.hikari.pool.HikariPool - Before shutdown pool stats HikariPoo
l-0 (total=10, inUse=3, avail=7, waiting=0)
[DEBUG] com.zaxxer.hikari.pool.HikariPool - After shutdown pool stats HikariPool
-0 (total=0, inUse=0, avail=0, waiting=0)
[INFO] com.fg.re.config.CacheConfiguration - Remove Cache Manager metrics
[INFO] com.fg.re.config.CacheConfiguration - Closing Cache Manager
[INFO] com.hazelcast.core.LifecycleService - [192.168.1.9]:5701 [dev] [3.2.5] Ad
dress[192.168.1.9]:5701 is SHUTTING_DOWN
[INFO] com.hazelcast.initializer - [192.168.1.9]:5701 [dev] [3.2.5] Destroying n
ode initializer.
[INFO] com.hazelcast.instance.Node - [192.168.1.9]:5701 [dev] [3.2.5] Hazelcast
Shutdown is completed in 31 ms.
[INFO] com.hazelcast.core.LifecycleService - [192.168.1.9]:5701 [dev] [3.2.5] Ad
dress[192.168.1.9]:5701 is SHUTDOWN
[ERROR] org.springframework.boot.SpringApplication - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean wit
h name 'mbeanExporter' defined in class path resource [org/springframework/conte
xt/annotation/MBeanExportConfiguration.class]: Invocation of init method failed;
 nested exception is org.springframework.jmx.export.UnableToRegisterMBeanExcepti
on: Unable to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSou
rce'; nested exception is javax.management.InstanceAlreadyExistsException: com.z
axxer.hikari:name=dataSource,type=HikariDataSource
        at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.initializeBean(AbstractAutowireCapableBeanFactory.java:1554) ~[spring-be
ans-4.0.8.RELEASE.jar:4.0.8.RELEASE]

Fresh observation. Earlier I had two jhipster webapps running on the same tomcat. Removing the other one has taken off the error. Can I know the reason? In production there can be a high possibility of two jhipster webapps running on the same tomcat.

Answer 1

I was having a similar issue, with 2 jhipster application instances running alongside on a single tomcat server. I posted this also in https://github.com/jhipster/generator-jhipster/issues/874#issuecomment-113023849

From https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html

Every JMX MBean must have an object name. The object name is an instance of the JMX class ?ObjectName and must conform to the syntax defined by the JMX specification. Namely, the object name must contain a domain and a list of key-properties.

From http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#boot-features-jmx

36. Monitoring and management over JMX Java Management Extensions (JMX) provide a standard mechanism to monitor and manage applications. By default Spring Boot will create an MBeanServer with bean id ‘mbeanServer’ and expose any of your beans that are annotated with Spring JMX annotations (@ManagedResource, @ManagedAttribute, @ManagedOperation).

See the JmxAutoConfiguration class for more details.

Checking the code of JmxAutoConfiguration in https://github.com/spring-projects/spring-boot/blob/master/spring-boot-autoconfigure/src/main/java/org/springframework/boot/autoconfigure/jmx/JmxAutoConfiguration.java I saw that there is a spring.jmx.default-domain property that is used in the naming strategy.

Setting that property to some value in one of my apps' application.properties solved this issue

spring.jmx.default-domain: test

As the domain name is arbitrary, it seems to me like a reasonable way to avoid name clashings between two apps.

Since I had no previous experience with JMX I would appreciate feedback on this solution.

Answer 2

I solved this kind of problem by setting different value of spring.jmx.default-domain property for each Spring applications on my Tomcat like:

spring.jmx.default-domain=somevalue

in application.properties file.

Answer 3

Two things I tried and they worked for deploying two Hikari apps in the same tomcat are: Not only change poolName(1) but also change bean name for DataSource(2) Configuration.

@Bean(destroyMethod = "shutdown")
public DataSource dataSource2() {

    HikariConfig config = new HikariConfig();
    config.setPoolName("AARSHikaripool-1");

Notice bean name is datasource2 while other app has datasource!!

Update for JHipster:

Since there is no datasource bean anymore, in application.yml add the following:

spring:
    jmx:
        default-domain: [application_name]

Answer 4

Simply add below annotation on your Config file

@EnableMBeanExport(registration=RegistrationPolicy.IGNORE_EXISTING)

It will work for you.

Answer 5

for me in tomcat 9 having multiple application and domain mapped. None of the above worked and had to improvise with

spring.jmx.default-domain=somevalue${random.uuid}

in application.properties

Answer 6

This seems like your application is re-deploying, but when it un-deployed the container did not call the close() or shutdown() method on the HikariDataSource. Spring should have a "destroy" property (or something akin) that can be set for un-deployment.

Also, make sure that you are using the latest version of HikariCP (2.2.5) if possible, I believe an old version did not unregister MBeans properly.

EDIT: if you have two WARs in the same VM that need HikariCP, and you want to register MBeans, you need to set each one to use a different poolName. I see it is using the default pool name of HikariPool-0.

Answer 7

The following approach worked for me, randomize the pool name jmx domain (spring.jmx.default-domain)(please see JmxAutoConfiguration) :

@Bean
public DataSource dataSource() throws SQLException {
    HikariDataSource dataSource = new HikariDataSource(this);
    dataSource.setPoolName("dataSource_" + UUID.randomUUID().toString());
    return dataSource;
}

@Bean
@ConditionalOnMissingBean(value = ObjectNamingStrategy.class, search = SearchStrategy.CURRENT)
public ParentAwareNamingStrategy objectNamingStrategy() {
    ParentAwareNamingStrategy namingStrategy = new ParentAwareNamingStrategy(new AnnotationJmxAttributeSource());
    namingStrategy.setDefaultDomain("domain_" + UUID.randomUUID().toString());
    return namingStrategy;
}