XWiki Security Cache global read/write lock scalability

classic Classic list List threaded Threaded
2 messages Options
ktc
Reply | Threaded
Open this post in threaded view
|

XWiki Security Cache global read/write lock scalability

ktc
We ran into an issue recently where users were not able to access the Wiki
due to the XWiki security cache holding up access to read requests as we
were experiencing heap exhaustion. A thread dump of a host showed that there
were 211 security cache read locks and 2 security cache write locks during
the event. Read locks are supposedly parallelizable, so we suspect that it
was the write locks that blocked the read requests.

Looking into the write locks, we found
*DefaultSecurityCacheRulesInvalidatorListener* shares a read-write lock
(*SecurityCacheRulesInvalidator*) with the security cache loader
(*DefaultSecurityCacheLoader*). It appears that these write requests to the
security cache were potentially creating a deadlock, blocking any further
read requests.

Our investigation of this event resulted in the following questions that we
are hoping to get answered by reaching out here:

1. What exactly happens in the event of a write failure to the Security
Cache? Are we correct in understanding that the write request will continue
to hold a lock on the cache, preventing all read requests from proceeding?
2. If so, are there any mitigations around the global read-write lock on the
security cache? Or, perhaps, is the best action is to simply move toward a
more scalable caching mechanism (i.e. a centralized cache such as Elastic
Cache)?
3. It looks as if all threads doing a read/write request to the
SecurityCache were all parked waiting for the same thread (3081f946);
however, the thread is not showing up in the thread dump so we are not sure
what it is stuck on. Does the XWiki team have any suspicions as to what
could be hold up the requests to the SecurityCache?

Thank you!

---

For more information on the event:

From our investigation, we concluded that the outage happened because:

1. Remote events triggered a security cache update and requested a couple of
write locks
2. The writes could not execute (perhaps due to the heap exhaustion,
although root cause of write failure is unknown)
3. ReentrantReadWriteLock lock followed a fair policy, which releases locks
in time order, so read requests were forced to wait for any previous write
requests to finish. Because the write requests couldn't finish, the
remaining 211 threads were blocked.

* Possible remote events that trigger security cache update: changes to an
XWiki group such as leaving the admin group, user first time visiting XWiki,
etc..

----

*LDAP Connection Lock example:*
 
Thread t@225108
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Object.wait(Native Method)
        - waiting on <7249fc39> (a com.novell.ldap.Connection)
        at com.novell.ldap.Connection.waitForReader(Unknown Source)
        at com.novell.ldap.Connection.startReader(Unknown Source)
        at com.novell.ldap.Connection.connect(Unknown Source)
        at com.novell.ldap.Connection.connect(Unknown Source)
        at com.novell.ldap.LDAPConnection.connect(Unknown Source)
        at
com.ownership.groups.ldap.DefaultLdapConnection.connect(DefaultLdapConnection.java:25)
        at
com.ownership.groups.ldap.LdapClient.queryGroups(LdapClient.java:99)
        at
com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:64)
        at
com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:59)
        at
com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:113)
        - locked <7cc1cfca> (a java.lang.String)
        at
com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:75)
        at
org.xwiki.security.internal.DefaultUserBridge.getGroupsReferencesFor(DefaultUserBridge.java:122)
        at
org.xwiki.security.internal.DefaultUserBridge.getAllGroupsFor(DefaultUserBridge.java:78)
        at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:350)
        at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:342)
        at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserEntry(DefaultSecurityCacheLoader.java:318)
        at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadAccessEntries(DefaultSecurityCacheLoader.java:221)
        at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadRequiredEntries(DefaultSecurityCacheLoader.java:193)
        at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:148)
        at
org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
        at
org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
        at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
        at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109)
        at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.displayTitle(AbstractDocumentTitleDisplayer.java:163)
        at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:149)
        at
org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:58)
        at
org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:96)
        at
org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:39)
        at
org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:123)
        at
org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:52)
        at
org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:68)
        at
org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:42)
        at
com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1538)
        at
com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1558)
        at com.xpn.xwiki.api.Document.getDisplayTitle(Document.java:345)
        at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.velocity.runtime.parser.node.PropertyExecutor.execute(PropertyExecutor.java:142)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:539)
        at
org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:198)
        at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at
org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
        at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
        at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.apache.velocity.runtime.parser.node.ASTStringLiteral.value(ASTStringLiteral.java:330)
        at
org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
        at
org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
        at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
        at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
        at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
        at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
        at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
        at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
        at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
        at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
        at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
        at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
        at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
        at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
        at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
        at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463)
        at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210)
        at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
        at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
        at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
        at
org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215)
        at
com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176)
        at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
        at
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        - locked <1cc68e56> (a org.apache.tomcat.util.net.SocketWrapper)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - locked <79120011> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
 
----
 
*Security Cache Read Lock example:*
 
Thread t@225106
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <3081f946> (a
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at
org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidator.suspend(DefaultSecurityCacheRulesInvalidator.java:56)
        at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:144)
        at
org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
        at
org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
        at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
        at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109)
        at
org.xwiki.security.authorization.internal.XWikiCachingRightService.hasAccessLevel(XWikiCachingRightService.java:309)
        at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:270)
        at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:254)
        at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
        at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
        at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at
org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
        at
org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
        at
org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
        at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
        at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
        at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
        at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
        at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
        at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
        at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
        at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
        at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
        at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
        at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
        at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
        at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
        at
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
        at
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
        at
org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
        at
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
        at
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
        at
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
        at
org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
        at
com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
        at
com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
        at
com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
        at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
        at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463)
        at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210)
        at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
        at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
        at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
        at
org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215)
        at
com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176)
        at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
        at
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        - locked <4da86584> (a org.apache.tomcat.util.net.SocketWrapper)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - locked <1479d987> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
 
----
 
*Security Cache Write Lock example:*
 
Thread t@178
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <3081f946> (a
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
        at
org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidatorListener.onEvent(DefaultSecurityCacheRulesInvalidatorListener.java:186)
        at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:304)
        at
org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:269)
        at
org.xwiki.observation.remote.internal.DefaultRemoteObservationManager.notify(DefaultRemoteObservationManager.java:168)
        at
com.wiki.observation.remote.jgroups.JGroupsReceiver.receive(JGroupsReceiver.java:87)
        at org.jgroups.JChannel.up(JChannel.java:768)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1034)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:438)
        at org.jgroups.stack.Protocol.up(Protocol.java:426)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:487)
        at
org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:989)
        at
org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:919)
        at
org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:851)
        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611)
        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
        at org.jgroups.protocols.FD.up(FD.java:260)
        at org.jgroups.protocols.MERGE3.up(MERGE3.java:292)
        at org.jgroups.protocols.Discovery.up(Discovery.java:296)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1657)
        at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1873)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - locked <2ea19041> (a
java.util.concurrent.ThreadPoolExecutor$Worker)



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html


Reply | Threaded
Open this post in threaded view
|

Re: XWiki Security Cache global read/write lock scalability

vmassol
Administrator
Hi Ktc,

Thanks for this detailed email with lots of details and hard questions!

This is not an easy mail to answer and I certainly don’t have the answers.

You’re the first one to report such issue AFAIK and that’s most certainly due to the scale at which you use XWiki :)

Let’s see if someone has the answer here. If you don’t get any answer, know that there’s an option for you which is to contract a company offering dedicated professional support for XWiki and have them spend quality time debugging this with you (see http://www.xwiki.org/xwiki/bin/view/Main/Support#HProfessionalSupport).

If you find the solution, some Pull Requests would be much appreciated to fix the problems.

Thanks
-Vincent


> On 25 Oct 2017, at 18:31, ktc <[hidden email]> wrote:
>
> We ran into an issue recently where users were not able to access the Wiki
> due to the XWiki security cache holding up access to read requests as we
> were experiencing heap exhaustion. A thread dump of a host showed that there
> were 211 security cache read locks and 2 security cache write locks during
> the event. Read locks are supposedly parallelizable, so we suspect that it
> was the write locks that blocked the read requests.
>
> Looking into the write locks, we found
> *DefaultSecurityCacheRulesInvalidatorListener* shares a read-write lock
> (*SecurityCacheRulesInvalidator*) with the security cache loader
> (*DefaultSecurityCacheLoader*). It appears that these write requests to the
> security cache were potentially creating a deadlock, blocking any further
> read requests.
>
> Our investigation of this event resulted in the following questions that we
> are hoping to get answered by reaching out here:
>
> 1. What exactly happens in the event of a write failure to the Security
> Cache? Are we correct in understanding that the write request will continue
> to hold a lock on the cache, preventing all read requests from proceeding?
> 2. If so, are there any mitigations around the global read-write lock on the
> security cache? Or, perhaps, is the best action is to simply move toward a
> more scalable caching mechanism (i.e. a centralized cache such as Elastic
> Cache)?
> 3. It looks as if all threads doing a read/write request to the
> SecurityCache were all parked waiting for the same thread (3081f946);
> however, the thread is not showing up in the thread dump so we are not sure
> what it is stuck on. Does the XWiki team have any suspicions as to what
> could be hold up the requests to the SecurityCache?
>
> Thank you!
>
> ---
>
> For more information on the event:
>
> From our investigation, we concluded that the outage happened because:
>
> 1. Remote events triggered a security cache update and requested a couple of
> write locks
> 2. The writes could not execute (perhaps due to the heap exhaustion,
> although root cause of write failure is unknown)
> 3. ReentrantReadWriteLock lock followed a fair policy, which releases locks
> in time order, so read requests were forced to wait for any previous write
> requests to finish. Because the write requests couldn't finish, the
> remaining 211 threads were blocked.
>
> * Possible remote events that trigger security cache update: changes to an
> XWiki group such as leaving the admin group, user first time visiting XWiki,
> etc..
>
> ----
>
> *LDAP Connection Lock example:*
>
> Thread t@225108
>   java.lang.Thread.State: TIMED_WAITING
>        at java.lang.Object.wait(Native Method)
>        - waiting on <7249fc39> (a com.novell.ldap.Connection)
>        at com.novell.ldap.Connection.waitForReader(Unknown Source)
>        at com.novell.ldap.Connection.startReader(Unknown Source)
>        at com.novell.ldap.Connection.connect(Unknown Source)
>        at com.novell.ldap.Connection.connect(Unknown Source)
>        at com.novell.ldap.LDAPConnection.connect(Unknown Source)
>        at
> com.ownership.groups.ldap.DefaultLdapConnection.connect(DefaultLdapConnection.java:25)
>        at
> com.ownership.groups.ldap.LdapClient.queryGroups(LdapClient.java:99)
>        at
> com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:64)
>        at
> com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:59)
>        at
> com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:113)
>        - locked <7cc1cfca> (a java.lang.String)
>        at
> com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:75)
>        at
> org.xwiki.security.internal.DefaultUserBridge.getGroupsReferencesFor(DefaultUserBridge.java:122)
>        at
> org.xwiki.security.internal.DefaultUserBridge.getAllGroupsFor(DefaultUserBridge.java:78)
>        at
> org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:350)
>        at
> org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:342)
>        at
> org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserEntry(DefaultSecurityCacheLoader.java:318)
>        at
> org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadAccessEntries(DefaultSecurityCacheLoader.java:221)
>        at
> org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadRequiredEntries(DefaultSecurityCacheLoader.java:193)
>        at
> org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:148)
>        at
> org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
>        at
> org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
>        at
> org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
>        at
> org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109)
>        at
> org.xwiki.display.internal.AbstractDocumentTitleDisplayer.displayTitle(AbstractDocumentTitleDisplayer.java:163)
>        at
> org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:149)
>        at
> org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:58)
>        at
> org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:96)
>        at
> org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:39)
>        at
> org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:123)
>        at
> org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:52)
>        at
> org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:68)
>        at
> org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:42)
>        at
> com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1538)
>        at
> com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1558)
>        at com.xpn.xwiki.api.Document.getDisplayTitle(Document.java:345)
>        at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:498)
>        at
> org.apache.velocity.runtime.parser.node.PropertyExecutor.execute(PropertyExecutor.java:142)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:539)
>        at
> org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:198)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
>        at
> org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.apache.velocity.runtime.parser.node.ASTStringLiteral.value(ASTStringLiteral.java:330)
>        at
> org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
>        at
> org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
>        at
> com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
>        at
> com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
>        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
>        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
>        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
>        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:498)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
>        at
> org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
>        at
> org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
>        at
> com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
>        at
> com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
>        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
>        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
>        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
>        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:498)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
>        at
> org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
>        at
> org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
>        at
> com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
>        at
> com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
>        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
>        at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
>        at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463)
>        at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210)
>        at
> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
>        at
> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
>        at
> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
>        at
> org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215)
>        at
> com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176)
>        at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
>        at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
>        at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
>        at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
>        at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>        at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157)
>        at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
>        at
> org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
>        at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>        at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
>        at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126)
>        at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
>        at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
>        - locked <1cc68e56> (a org.apache.tomcat.util.net.SocketWrapper)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>        at java.lang.Thread.run(Thread.java:748)
>
>   Locked ownable synchronizers:
>        - locked <79120011> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> ----
>
> *Security Cache Read Lock example:*
>
> Thread t@225106
>   java.lang.Thread.State: WAITING
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for <3081f946> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
>        at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>        at
> org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidator.suspend(DefaultSecurityCacheRulesInvalidator.java:56)
>        at
> org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:144)
>        at
> org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
>        at
> org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
>        at
> org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
>        at
> org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109)
>        at
> org.xwiki.security.authorization.internal.XWikiCachingRightService.hasAccessLevel(XWikiCachingRightService.java:309)
>        at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:270)
>        at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:254)
>        at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:498)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
>        at
> org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
>        at
> org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
>        at
> org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
>        at
> com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
>        at
> com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
>        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
>        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
>        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
>        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:498)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
>        at
> org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
>        at
> org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
>        at
> com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
>        at
> com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
>        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
>        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
>        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
>        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:498)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
>        at
> org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
>        at
> org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
>        at
> com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
>        at
> com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
>        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
>        at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080)
>        at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886)
>        at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:498)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
>        at
> org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
>        at
> org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
>        at
> org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311)
>        at
> org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230)
>        at
> org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
>        at
> org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
>        at
> org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259)
>        at
> org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222)
>        at
> com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616)
>        at
> com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602)
>        at
> com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77)
>        at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102)
>        at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180)
>        at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463)
>        at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210)
>        at
> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
>        at
> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
>        at
> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
>        at
> org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215)
>        at
> com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176)
>        at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>        at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
>        at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
>        at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
>        at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
>        at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>        at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157)
>        at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
>        at
> org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
>        at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>        at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
>        at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126)
>        at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
>        at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
>        - locked <4da86584> (a org.apache.tomcat.util.net.SocketWrapper)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>        at java.lang.Thread.run(Thread.java:748)
>
>   Locked ownable synchronizers:
>        - locked <1479d987> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> ----
>
> *Security Cache Write Lock example:*
>
> Thread t@178
>   java.lang.Thread.State: WAITING
>        at sun.misc.Unsafe.park(Native Method)
>        - parking to wait for <3081f946> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>        at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
>        at
> org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidatorListener.onEvent(DefaultSecurityCacheRulesInvalidatorListener.java:186)
>        at
> org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:304)
>        at
> org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:269)
>        at
> org.xwiki.observation.remote.internal.DefaultRemoteObservationManager.notify(DefaultRemoteObservationManager.java:168)
>        at
> com.wiki.observation.remote.jgroups.JGroupsReceiver.receive(JGroupsReceiver.java:87)
>        at org.jgroups.JChannel.up(JChannel.java:768)
>        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1034)
>        at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
>        at org.jgroups.protocols.FlowControl.up(FlowControl.java:438)
>        at org.jgroups.stack.Protocol.up(Protocol.java:426)
>        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
>        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:487)
>        at
> org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:989)
>        at
> org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:919)
>        at
> org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:851)
>        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611)
>        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
>        at org.jgroups.protocols.FD.up(FD.java:260)
>        at org.jgroups.protocols.MERGE3.up(MERGE3.java:292)
>        at org.jgroups.protocols.Discovery.up(Discovery.java:296)
>        at org.jgroups.protocols.TP.passMessageUp(TP.java:1657)
>        at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1873)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at java.lang.Thread.run(Thread.java:748)
>
>   Locked ownable synchronizers:
>        - locked <2ea19041> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
>
>
> --
> Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html
>
>