Tomcat websocket code succeeds on localhost but fails when hosted and modified for remote host

This has been discussed before, but never successfully (as far as I can tell) resolved. I'm using Tomcat 9, eclipse, and of course Java and Javascript.

Chapter 1

I can make work neither the well-written javaCodeGeeks websocket example, or for the purposes of providing a minimal framework, re-hosting the Tomcat example chat/echo code outside of their magic realm. (I'll have more to say about this below.)

I've also been poring over the Tomcat logs and scrutinizing the GET requests which return 404 errors and mystery 7xx error codes, some of which I've been able to infer. (There is no general documentation of the 7xx codes.) I'll discuss those at the bottom.

So the nub of the issue is that Websocket code works end-to-end when hosted by Tomcat on localhost, but when hosted on a remote host, the connection created by "new WebSocket" fails. (This issue has been discussed several times, in multiple forums, but it seems more likely that the questioners have given up rather than succeeded and moved on.)

So I would like to know two things. First, how to make my example code work when hosted remotely (any port in a storm) and second, I would like to know how the tomcat examples work at all. The sample I've concocted is not something new and suspect; it is simply the chat annotation from the Tomcat examples. (There are a few reductions so that it works in a minimal environment, and those are noted. In particular the logging routines, and a parse-html class was pulled from its home on Tomcat.)

The result of this sample when executed against a remote server is a 404 error (with some curious 7xx error codes, which I observe and report on below).

If you want to cut to the code, here is a link to a war file that includes sources; They are just adopted from the Tomcat sample chat server to run in a minimum configuration. Three files, 2 java and one index.html. You can run/deploy it directly on tomcat in webapps or pull the sources into eclipse to compare and edit.

SampleCode
Dossier

https://1drv.ms/u/s!AqbqlA2vzkjRcrGjW8rDRLW3o50?e=fgeh0U

Here is the point:

   var target;
   // Shows the bug: unless location resolves to local host won't work
       	target = 'ws://' + window.location.host + 
       '/SocketBugMinimum/websocket/chat';
		//
		// For testing purposes,
        // access the tomcat one in the magic directory, not this one...
        //target = 'ws://' + window.location.host +      
        //     '/examples/websocket/chat';
        // also try:
        //    examples/websocket/echoStreamAnnotation
        // it will work too...

And when it goes behind the curtain

    Chat.socket = new WebSocket(target);

it returns with the error on the console:
> WebSocket connection to ws:REMOTEHOST:8080/SocketBugMinimum/websocket/chat failed: Error during WebSocket handshake: Unexpected response code: 404

Now a couple of things to note.

1. Works fine when host is localhost.
2. You can connect to remote host accessing the Tomcat examples websocket, even though they have nothing to do with this code. This is both fun and instructive.
3. This works even when you are yourself hosted on localhost.

The SocketBugMinimun app produces the same result run from eclipse, accessing localhost or remotehost from either chrome or firefox, and from the tomcat html-manager (i.e. host...8080:/Manager/html).

Now looking through the Tomcat logfile (grep the ones with today's date for GET to see all the connect activity) you will find an interesting numeric code beside the reported 404 error.

As I promised above, I've tried to infer meanings of some of the 7xx codes produced in the logfile; they seem to modify the 404 error. (7xx codes are not part of HTTP as reported in an RFC; they seem to be recycled and augmented Modem codes, of all things.) If anyone has a better understanding, please share it. Tomcat-developer endorsed description of these codes it would be great to know. apparently tomcat websockets busily produce different and rich errors, but there is just no way to know what they are. (I've scoured the tomcat docs, and Google finds nothing.) In order of usefulness:

727 Listener exists but terminal endpoint is bad
733 Endpoint exists and is hunky dory but I won't let you connect to it because I'm mean
719 No root element
746 Valid root, but the rest of the path is bad
726 You tried /a/b but you need /a/b/c
737 Beats me
748 No idea

Chapter 2

Now the mystery to me is how the Tomcat examples work at all.

It appears that by being hosted in the examples directly they seem to have conferred on them some magical powers to use sockets. I've tried to reproduce those results using catalina privileges but am unsuccessful. (And if you are a connoisseur of Tomcat privileges you will note that the sample files says "this is what you should do to generate these effects" but those lines are COMMENTED OUT and the examples work anyway!)

I also experimented with Valves, but have nothing useful to report. If it wouldn't kill the Tomcat people to provide a few more documented examples, it would be helpful.

There is also a voodoo file in Tomcat called servers.json (you will note that no other files with a .json extension are shipped with the product). This file and its curious and undocument incantations change the names of the network pathname to the endpoints. (This file is why I chose to repro this with chat, which uses no other magic, rather than echo.)

One likely solution might be that the examples are conferred special powers in catalina.policy. But no dice. I tried to add WebSocket privs, and it doesn't do any good. Here is the diff between the original and what I added.

 // ++
< // everyone needs sockets!
< grant {
<    permission java.net.SocketPermission "*:8080", "accept,connect,listen,resolve";
< };
< 
< // Granting permission for Spades to be reached via websocket
< grant codeBase "war:file:${catalina.base}/webapps/Spades.war*/-" {
<    permission java.net.SocketPermission "*:8080", "accept,connect,listen,resolve";
<    permission java.lang.RuntimePermission "accessClassInPackage.org.apache.tomcat.websocket";
<    permission java.lang.RuntimePermission "accessClassInPackage.org.apache.tomcat.websocket.server";
<    permission java.sercurity.AllPermission;
< };
< //--
< 
275c260
< // };

As I said, it didn't seem to do any good. Same sad errors in the logfiles.

Thanks in advance for your help.
Marklan

Problem solved.

I did some more snooping around and learned that there are discrepancies in Tomcat results when you are running older versions of Java. Coderanch had this helpful article:

    https://coderanch.com/t/659638/application-servers/WebSocket-deploying-Server

I checked and realized I had Java 1.8 on my server. (Dooh!)

Thanks to ServerWorld for its helpful site, here is helpful information on upgrading Java, finding the version you need, etc. Here is a link to the English version, from there you can select OS config, etc.

    https://www.server-world.info/en/

So, I upgraded to version 11 JDK (i.e. downloading jdk-11.0.1_linux-x64_bin.rpm from oracle.com) running rpm to install, i.e. from the download directory

    rpm -Uvh jdk-11*

to install.
Finally configuring away from the old version without deleting it (in case this doesn't work!)

    alternatives --config java

to select and install the latest. Restarted Tomcat and it all worked.
Thanks to the people who looked at this with me, and scratched their heads. I appreciated the efforts.

Happy coding. Hopefully this mis-adventure can be helpful to anyone else who has the same problem.

And it would sure be nice if the Tomcat team could document the 7xx errors. I have no idea why the older versions of Java did not work. I can't imagine how something like this would be version dependent, but I guess my imagination is just not that good...

I did find some other useful stuff that I want to share. In particular the magic cookie that makes the examples files work is buried in a call to ServerApplicationConfig, wheich essentially exports and deploys the endpoints in subdirectories. Check out the file ExamplesConfig.java buried off in webapps/WEB-INF/classes/websocket. It takes advantage of a Tomcat hook that scans your subdirectories for you, and exports endpoints, or declines to export them as you wish.

M