Just to let you know, today I’ve been trying to join a network (network ID 8417) and it is failing by timeout.
I see the husarnet join fails by timeout after 1 minute, so I tried using Husarnet sidecar container, and after more than 10 minutes of trying, it eventually connected.
Just for you to know, here are the details of when it was failing to connect (when connected, the IP was also fc94:449c:0b65:a4f1:64a4:dec2:2f91:182a)
root@1_xterm:/# husarnet status
Version
🟢 CLI: 2.0.0
🟢 Daemon (running): 2.0.0
🟢 Daemon (binary): 2.0.0
Dashboard URL
🟢 CLI: app.husarnet.com
🟢 Daemon: app.husarnet.com
Connection status
🟢 Base Server: 188.165.23.196:5582 (UDP)
Readiness
🟢 Is ready to handle data? yes
🟢 Is ready to join? yes
🔴 Is joined? no
Local
⚪ Husarnet IP:
fc94:449c:0b65:a4f1:64a4:dec2:2f91:182a
Whitelist
fc94:b01d:1803:8dd8:b293:5c7d:7639:932a (websetup)
🟢 active 🟢 secure 🟡 tunelled
As I said, it eventually connected, so, this post is more to let you be aware that sometimes this long waiting for a connection happens.
My docker container is running --net host --privileged and below are the logs
husarnet-daemon & (sleep 10 && husarnet join fc94:b01d:1803:8dd8:b293:5c7d:7639:932a/XXXXXXXXXXXXXXXXXXX vpn-01)
[1] 643191
WARNING allocated tun hnet0 (/app/core/ports/unix/tun.cpp:44)
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.hnet0.disable_ipv6 = 0
WARNING ngsocket fc94:59d8:c2b9:c7de:8bab:76c8:a7b5:ba31 listening on 5582 (/app/core/ngsocket.cpp:580)
INFO Sending update request to websetup (/app/core/websetup.cpp:155)
WARNING waiting until base address is resolved... (/app/core/ngsocket.cpp:887)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
WARNING send multicast to 1 destinations (/app/core/multicast_layer.cpp:106)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
WARNING HTTP thread bound to 127.0.0.1:16216. Will start handling the connections. (/app/core/api_server/server.cpp:361)
WARNING waiting until base address is resolved... (/app/core/ngsocket.cpp:887)
WARNING waiting until base address is resolved... (/app/core/ngsocket.cpp:887)
WARNING retrying with fallback base address: 188.165.23.196:443 (/app/core/ngsocket.cpp:882)
INFO establishing connection to base 188.165.23.196:443 (/app/core/ngsocket.cpp:891)
WARNING TCP connection to base server established (/app/core/ngsocket.cpp:395)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
WARNING send multicast to 1 destinations (/app/core/multicast_layer.cpp:106)
INFO hello 2 from fc94b01d18038dd8b2935c7d7639932a (/app/core/security_layer.cpp:165)
WARNING established secure connection to fc94b01d18038dd8b2935c7d7639932a (/app/core/security_layer.cpp:254)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
INFO Sending update request to websetup (/app/core/websetup.cpp:155)
WARNING tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT (/app/core/ngsocket.cpp:838)
WARNING UDP connection to base server established (/app/core/ngsocket.cpp:370)
WARNING saving settings (/app/core/config_storage.cpp:78)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
SUCCESS: Successfully registered a join request
SUCCESS: Waiting for Base server connection (any protocol)…
SUCCESS: Waiting for websetup connection…
▄ Waiting until the device is joined… (401ms)WARNING send multicast to 1 destinations (/app/core/multicast_layer.cpp:106)
▀ Waiting until the device is joined… (4.813s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
▀ Waiting until the device is joined… (9.827s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▀ Waiting until the device is joined… (14.64s)WARNING send multicast to 1 destinations (/app/core/multicast_layer.cpp:106)
▄ Waiting until the device is joined… (14.841s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▄ Waiting until the device is joined… (18.852s)WARNING tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT (/app/core/ngsocket.cpp:838)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
▄ Waiting until the device is joined… (19.855s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
▀ Waiting until the device is joined… (24.869s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▀ Waiting until the device is joined… (29.882s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▀ Waiting until the device is joined… (34.694s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▄ Waiting until the device is joined… (39.707s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▄ Waiting until the device is joined… (43.116s)WARNING send multicast to 1 destinations (/app/core/multicast_layer.cpp:106)
▀ Waiting until the device is joined… (44.119s)WARNING tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT (/app/core/ngsocket.cpp:838)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
▄ Waiting until the device is joined… (44.72s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▀ Waiting until the device is joined… (45.121s)WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
▀ Waiting until the device is joined… (49.733s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▀ Waiting until the device is joined… (54.748s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
▄ Waiting until the device is joined… (59.761s)WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
ERROR: timeout
root@shark-desktop:/ran# WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT (/app/core/ngsocket.cpp:838)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT (/app/core/ngsocket.cpp:838)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING send multicast to 1 destinations (/app/core/multicast_layer.cpp:106)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING Sending join request to websetup (/app/core/websetup.cpp:150)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT (/app/core/ngsocket.cpp:838)
INFO reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] (/app/core/ngsocket.cpp:232)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
WARNING peer fc941dc779b8e2fd9df8b003091158fa is blacklisted (/app/core/peer_container.cpp:36)
I think the key line is reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] it doens´t connect with the websetup server?
The issue is sadly sporadic. On my laptop this works fine and it joins within seconds. Now im trying to set things up on multiple PCs here at the office for a test, but only my laptop was able to register and a collegues laptop after 5-10 mins. Now im working on a desktop and I’ve yet to made it to be able to connect. Also running it at home from my home desktop didnt work for around an hour, and then it suddenly worked without me having done anything special.
This is the husarnet status output:
root@vpn-01:/ran# husarnet status
Version
● CLI: 2.0.13
● Daemon (running): 2.0.13
● Daemon (binary): 2.0.13
Dashboard URL
● CLI: app.husarnet.com
● Daemon: app.husarnet.com
Connection status
● Base Server: 188.165.23.196:5582 (UDP)
Readiness
● Is ready to handle data? yes
● Is ready to join? yes
● Is joined? no
Local
● Husarnet IP:
fc94:59d8:c2b9:c7de:8bab:76c8:a7b5:ba31
Whitelist
fc94:b01d:1803:8dd8:b293:5c7d:7639:932a (websetup)
● active ● secure ● tunelled
A final note, i tried clearing the /var/lib/husarnet folder but that didnt have any effect either.
Also we are, for now, still under the 5 free devices limit as we are still integrating this into our software, so it’s not a client limit either.
if these tests work, we will deploy husarnet on our drones.
Hi @deniz!
Thanks for a wonderful bug report. As per the sporadic websetup connection issues - we will dig deeper into this, but it’ll take more time. As per a temporary test - Husarnet 2.0 added a couple of environment variables to ease a development of integrations like yours. You can change the invocation to HUSARNET_HOSTNAME=vpn-01 HUSARNET_JOIN_CODE=XXXXXXXXXXXXXXXXXXX husarnet-daemon - this should make the daemon attempt a join on it’s own, with no extra scripts and delays required. (Also - you can drop the IPv6 address from join code if you like - the new version of Husrnet supports both short and long versions.) Also - have a look at husarnet daemon wait --help - the joinable option may be helpful to you if you decide not to go with a environment variable route.
thank you for this information! setting the env variables would definitely make things a bit cleaner.
As for the issue, I think we did hit the 5 nodes limit of the free plan but weren’t aware of it. I am unable now to reproduce it but what happened was that the dashboard’s /network tab showed 1 Network 0 Devices, but when i went to the elements it showed 5 old devices that we didnt use anymore.
A small work-around that did work was manually adding an element to a network. When I did this then i could see from the daemon that the join was successful, but it is unable to join itself the first time. Once i added it manually, it reconnects without a problem.
So i think the joining issue is somehow related to hitting the limit but then me not seeing that reflected in the dasboard/network window. Or it going wrong somewhere in the automatic joining section.
Either way, I am aware of this now. Maybe a return code / warning / when attempting to join a network specifying that the limit was reached, would be useful? Or a dashboard notification that some device attempted to join but was rejected, would help in such a case in the future?
Greetings and thanks again for your time. This product is amazing and it all works seemlessly now. I streamed GPS data from a drone at work to my laptop in a cafe a few days ago and that was super exciting.
I’m glad that you’ve found the issue and enoying the product! Adding a proper warning for quota limits is on our todo list for the longest time and one of our requirements for Husarnet 2.1, so it will be done in the next few months. Same goes for making sure that joining an already “enrolled” element/device will add it to proper network automatically.
Just a quick note: we’ve recently updated our docs regarding the Docker setup - including the environment variables I’ve mentioned earlier and suggestions on building Husarnet into your own images.
The URL may change in the future but right now it’s under: Launching Husarnet in a Docker container | Husarnet
If you have any suggestions/feedback/questions/… please let us know!