Getting io.netty.util.ResourceLeakDetector leak exception

[jrgp]

browserup proxy is so far looks good for us. During local testing I'm seeing this exception periodically though:

14:41:28.972 [LittleProxy-13-ClientToProxyWorker-5] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
	io.netty.buffer.CompositeByteBuf.allocBuffer(CompositeByteBuf.java:1837)
	io.netty.buffer.CompositeByteBuf.copy(CompositeByteBuf.java:1482)
	io.netty.buffer.AbstractByteBuf.copy(AbstractByteBuf.java:1187)
	io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.copy(HttpObjectAggregator.java:412)
	org.littleshoot.proxy.impl.ClientToProxyConnection.copy(ClientToProxyConnection.java:1040)
	org.littleshoot.proxy.impl.ClientToProxyConnection.doReadHTTPInitial(ClientToProxyConnection.java:210)
	org.littleshoot.proxy.impl.ClientToProxyConnection.readHTTPInitial(ClientToProxyConnection.java:186)
	org.littleshoot.proxy.impl.ClientToProxyConnection.readHTTPInitial(ClientToProxyConnection.java:54)
	org.littleshoot.proxy.impl.ProxyConnection.readHTTP(ProxyConnection.java:144)
	org.littleshoot.proxy.impl.ProxyConnection.read(ProxyConnection.java:122)
	org.littleshoot.proxy.impl.ProxyConnection.channelRead0(ProxyConnection.java:582)
	io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	org.littleshoot.proxy.impl.ProxyConnection$RequestReadMonitor.channelRead(ProxyConnection.java:709)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	org.littleshoot.proxy.impl.ProxyConnection$BytesReadMonitor.channelRead(ProxyConnection.java:686)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	java.lang.Thread.run(Thread.java:748)

Is this something you've seen before?

[ericbeland]

We'll take a look. This message probably came in with the Netty upgrade, and the leak has most likely been there all along. Netty 4 added reference counting and logging of detected leaks.

[ericbeland]

Reference on getting to the bottom of netty leaks: https://github.com/netty/netty/wiki/Reference-counted-objects

[jrgp]

I wonder if this is problem is buried within the littleproxy codebase rather than being a bug in browserup-proxy.

[ericbeland]

I think that is likely the case @jrgp as littleproxy brings in netty.

[dpowell]

@jrgp Do you have a test case to reproduce this?

[jrgp]

@dpowell I can't provide specific information of our internal usage of browserup-proxy and what resources we hit, but I'll give you a general overview:

I can repro this by using python selenium/python browsermob-proxy with "headless" Firefox hitting several pages that load lots of static assets as well as make lots of ajax calls.

Our usage pattern is: 1) make a new har 2) spawn firefox and make it use that har's proxy 3) hit a bunch of stuff 4) immediately tear down firefox and retrieve/kill the har, and this process happens several times a minute with a single long-lived browserup-proxy instance. We also have browserup-proxy go through a forward proxy rather than hit URLs directly.

My guess is once all that happens enough some resources get leaked at some point in that pipeline. Maybe it happens due to the fact we're making browserup-proxy go through an outgoing proxy and that codepath isn't tested often.

[ericbeland]

It looks like this issue most likely does live in littleproxy and is related to the chained proxy. We're evaluating some attempts at fixing the littleproxy issue in forked branches. More to come.

[jrgp]

Thanks for the update!

[ericbeland]

@dpowell There's a fix in the latest littleproxy for (at least one, maybe all) instances of this error.

[jrgp]

Hi @ericbeland any update on this?

[ericbeland]

Thanks for the reminder @jrgp. I think the change has been merged and the littleproxy release cut. I'll follow up on getting our new release out, hopefully tomorrow or Friday.

[ericbeland]

I haven't looked in depth personally, and can't guarantee that this will hit the case in question, but fingers crossed.

[ericbeland]

@jrgp We just cut a new release with an update to the latest LittleProxy. Let us know if it clears this error up for you or not. Cheers!