To debug Java processes, the JConsole is a good helper tool. It shows the Java process' internal memory usage as well as the internal threads and their thread stacks. But jconsole is only able to display information about the Java process, if the SSH user has the necessary permissions.
First connect to the machine on which your Java process is running. Enable X11 (display) forwarding with the additional -X parameter:
ck@mint:~$ ssh javaserver.example.com -X
On the server you can launch jconsole. If the JAVA_HOME path is in the $PATH variable, this should work. Otherwise you might have to find where jconsole is located and execute the full path:
ck@javaserver:~> /home/app/usr/java/jdk-11.0.14+9/bin/jconsole &
[1] 6883
This should fire up the jconsole GUI:
Unfortunately this only shows PID 6883, which is this very JConsole process firing up. However what I wanted to look at was the Java process of the application itself, running as PID 12123 under the user "app":
ck@javaserver:~> ps auxf|grep java
ck 6883 4.5 0.8 7966264 140200 pts/0 Sl 07:30 0:03 \_ /home/app/usr/java/jdk-11.0.14+9/bin/jconsole
ck 7541 0.0 0.0 10248 768 pts/0 S+ 07:31 0:00 \_ grep --color=auto java
app 12123 4.8 16.2 12212412 2635036 ? Sl Jun22 35:24 /home/app/usr/java/app/bin/java -Djava.awt.headless=true -Dcom.sun.management.jmxremote
Information about launched Java process is stored under /tmp/hsperfdata_USER. Looking closer into that directory, we can find a file named after our target PID (12123):
ck@javaserver:~> ls -la /tmp/hsperfdata_app/
total 52
drwxr-xr-x 2 app app 4096 Jun 22 19:21 .
drwxrwxrwt 73 root root 12288 Jun 23 07:31 ..
-rw------- 1 app app 32768 Jun 23 07:31 12123
Because the file itself is only readeable (and writeable) by the app user, my personal user (ck) is unable to read the Java process.
The root user can, of course, change the permissions:
root@javaserver:~ # chmod 644 /tmp/hsperfdata_digps/12123
After firing up jconsole once more with my own user, the Java process 12123 is now seen by JConsole, however it is still greyed out.
ck@javaserver:~> /home/app/usr/java/jdk-11.0.14+9/bin/jconsole &
[1] 8908
Looks like there are still additional permission errors. Instead of changing tons of application specific permissions, let's try something else.
Yes, a X/Display session can be transferred to another local user - as long as you can sudo or su to the other user. In this case I can become root on that javaserver. A (better) alternative would be to switch to the "app" user.
Each X session has an ID and a secret (like a password) associated with it. This can be listed with the xauth list command:
ck@javaserver:~> xauth list
javaserver/unix:10 MIT-MAGIC-COOKIE-1 a2645554d54ae8cd25d9c8ef55639699
Now we can su to another user and then "import" that X session to the other user using xauth add:
ck@javaserver:~> su -
Password: *********************
root@javaserver:~ # xauth add javaserver/unix:10 MIT-MAGIC-COOKIE-1 a2645554d54ae8cd25d9c8ef55639699
And now let's launch JConsole as another user (root in this case):
root@javaserver:~ # /home/app/usr/java/jdk-11.0.14+9/bin/jconsole &
[1] 22977
Eureka! Now we can select the wanted Java process (PID 12123) and are able to find more Java internal information about this process.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder