Friday, November 25, 2011

HotSpot (64bit server) hangs on socket read (JVM 1.7 bug?) - updated


Picture (c) Clker.com
Few days ago I started working on some maven-based project (Java 1.7.0) on my new laptop (with Windows 7 64bit installed) and I noticed that very often Maven 3 was stuck downloading JARs from the internet even though my internet connection was fine. I started investigating this issue and after checking thread dump and binary dump (DMP) I think I found a bug in HotSpot for 64bit Windows.


It's quite strange but any Java application using java.net.DualStackPlainSocketImpl (from rt.jar) class - JVM hangs on native socket0(boolean stream, boolean v6Only) method.

Here is an excerpt from the thread dump from my problematic Java app:

"main" prio=6 tid=0x0000000001d3b000 nid=0x6ec runnable [0x00000000020df000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000000eb31e490> (a java.io.BufferedInputStream)
at sun.net.www.MeteredStream.read(MeteredStream.java:134)
- locked <0x00000000eb321280> (a sun.net.www.http.KeepAliveStream)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2968)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000000eb2fcc08> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at io.DownloadJar.main(DownloadJar.java:20)

Locked ownable synchronizers:
- None

This is what I got from Win Debug Diagnostic Tool when I analyzed full memory / process image:
The following threads in java.dmp are waiting on data to be returned from another server via WinSock.

The call to WinSock originated from net!Java_java_net_SocketInputStream_socketRead0+160 ( 1 ) 5,56% of threads blocked

Ensure that any remote server this application may be calling is functioning properly and there are no network issues between the two servers. If the problem continues, please contact the application vendor for further assistance

Below you can find a source code of this simple app causing the problem (imports omitted):
// netty-3.2.6.Final.jar from maven repository
URL url = new URL("http://tinyurl.com/cg43cju");
try (BufferedInputStream in = new BufferedInputStream(url.openStream());
    BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream("netty.jar"))) {
   
 byte[] buf = new byte[2048];
 int len = 0;
 int total = 0;
 while ((len = in.read(buf)) > -1) {
  total += len;
  System.out.println(total / 1024 + "KB");
  out.write(buf, 0, len);
 }
} catch (IOException e) {
 e.printStackTrace();
}

This problem occurs very often (almost every time) and it's quite easy to reproduce (at least on my machine).

You would suggest to check my internet connection but it's fine. I'm able to download this JAR (or anything else) with no problems at all.

What is EXTREMELY weird is that if I run the very same byte code on HotSpot installed on Ubuntu that runs on VirtualBox on my Windows, it WORKS FINE 100% of time. WTF? I think it proves there is no problem with my network interface or connection but with HotSpot implementation for 64bit Windows?

If you encountered problems like this and know how to fix it, please let me know.

BTW. I already posted a bug report to Oracle but it's pending - I will provide a link here when it becomes public.

UPDATE
Bug link: 7115226

7 comments:

Anonymous said...

Doubt its your problem but the while loop should be using -1 instead of 0.

Przemysław Bielicki said...

It changes nothing but indeed it should be -1 - updated the code

thanks

Javin @ String split Java said...

Thanks for informing us dude. good work.

Thanks
Javin
Difference between ClassNotFoundException vs NoClassDefFoundError

Jörg Buchberger said...

Hi Przemysław

did you try
-Djava.net.preferIPv4Stack=true
and did that help anything?

Cheers

Przemysław Bielicki said...

@Jörg - yes, this is a workaround I posted as a comment to the bug report.

-Djava.net.preferIPv4Stack=true works fine

Thanks for your comment
Cheers

Игорь Потеряев said...

We had similar problem on windows vista machine and the reason was
TCP Window Scaling Auto Tuning.

You can try to disable it, as described here: http://www.mydigitallife.info/disable-tcp-auto-tuning-to-solve-slow-network-cannot-load-web-page-or-download-email-problems-in-vista/

Hope this helps.

Filip Hanik said...

I can reproduce this 100% of the times on a Windows 7 machine. It is definitely a bug in the JVM, as it is only reproducible with 1.7 (not 1.6 or 1.5)