Husarnet failing to connect (timeout after 1 minute, after 10+ minutes it connected)

Hi @ympek and @dominik,

Just to let you know, today I’ve been trying to join a network (network ID 8417) and it is failing by timeout.

I see the husarnet join fails by timeout after 1 minute, so I tried using Husarnet sidecar container, and after more than 10 minutes of trying, it eventually connected.

Just for you to know, here are the details of when it was failing to connect (when connected, the IP was also fc94:449c:0b65:a4f1:64a4:dec2:2f91:182a)

root@1_xterm:/# husarnet status 
Version
🟢 CLI:                       2.0.0
🟢 Daemon (running):          2.0.0
🟢 Daemon (binary):           2.0.0

Dashboard URL
🟢 CLI:                       app.husarnet.com
🟢 Daemon:                    app.husarnet.com

Connection status
🟢 Base Server:               188.165.23.196:5582 (UDP)

Readiness
🟢 Is ready to handle data?   yes
🟢 Is ready to join?          yes
🔴 Is joined?                 no

Local
⚪ Husarnet IP:
fc94:449c:0b65:a4f1:64a4:dec2:2f91:182a

Whitelist
fc94:b01d:1803:8dd8:b293:5c7d:7639:932a (websetup)
🟢 active     🟢 secure         🟡 tunelled

As I said, it eventually connected, so, this post is more to let you be aware that sometimes this long waiting for a connection happens.

Hello @ralves

Thank you so much for the report.
We’ll look into this.

Best regards,
Szymek

1 Like

We’ve just released 2.0.13 that hopefully has this issue fixed. Please do let us know if it’s indeed the case.


Have a wonderful day
Paweł

1 Like

Dear @pidpawel and @ympek ,

Thank you for your time looking into this. I think I am having the same issue. I compiled husarnet into my personal docker container and am running:

husarnet-daemon & (sleep 10 && husarnet join fc94:b01d:1803:8dd8:b293:5c7d:7639:932a/XXXXXXXXXXXXXXXXXXX vpn-01)

My docker container is running --net host --privileged and below are the logs

husarnet-daemon & (sleep 10 && husarnet join fc94:b01d:1803:8dd8:b293:5c7d:7639:932a/XXXXXXXXXXXXXXXXXXX vpn-01)
[1] 643191
WARNING  allocated tun hnet0                                                              (/app/core/ports/unix/tun.cpp:44)
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.hnet0.disable_ipv6 = 0
WARNING  ngsocket fc94:59d8:c2b9:c7de:8bab:76c8:a7b5:ba31 listening on 5582               (/app/core/ngsocket.cpp:580)
INFO     Sending update request to websetup                                               (/app/core/websetup.cpp:155)
WARNING  waiting until base address is resolved...                                        (/app/core/ngsocket.cpp:887)
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
WARNING  send multicast to 1 destinations                                                 (/app/core/multicast_layer.cpp:106)
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
WARNING  HTTP thread bound to 127.0.0.1:16216. Will start handling the connections.       (/app/core/api_server/server.cpp:361)
WARNING  waiting until base address is resolved...                                        (/app/core/ngsocket.cpp:887)
WARNING  waiting until base address is resolved...                                        (/app/core/ngsocket.cpp:887)
WARNING  retrying with fallback base address: 188.165.23.196:443                          (/app/core/ngsocket.cpp:882)
INFO     establishing connection to base 188.165.23.196:443                               (/app/core/ngsocket.cpp:891)
WARNING  TCP connection to base server established                                        (/app/core/ngsocket.cpp:395)
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
WARNING  send multicast to 1 destinations                                                 (/app/core/multicast_layer.cpp:106)
INFO     hello 2 from fc94b01d18038dd8b2935c7d7639932a                                    (/app/core/security_layer.cpp:165)
WARNING  established secure connection to fc94b01d18038dd8b2935c7d7639932a                (/app/core/security_layer.cpp:254)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
INFO     Sending update request to websetup                                               (/app/core/websetup.cpp:155)
WARNING  tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT      (/app/core/ngsocket.cpp:838)
WARNING  UDP connection to base server established                                        (/app/core/ngsocket.cpp:370)
WARNING  saving settings                                                                  (/app/core/config_storage.cpp:78)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
 SUCCESS:  Successfully registered a join request
 SUCCESS:  Waiting for Base server connection (any protocol)…                                                                                                                   
 SUCCESS:  Waiting for websetup connection…                                                                                                                                     
 ▄ Waiting until the device is joined…  (401ms)WARNING  send multicast to 1 destinations                                                 (/app/core/multicast_layer.cpp:106)    
▀  Waiting until the device is joined…  (4.813s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)          
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
 ▀ Waiting until the device is joined…  (9.827s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)          
 ▀ Waiting until the device is joined…  (14.64s)WARNING  send multicast to 1 destinations                                                 (/app/core/multicast_layer.cpp:106)   
 ▄ Waiting until the device is joined…  (14.841s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
 ▄ Waiting until the device is joined…  (18.852s)WARNING  tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT      (/app/core/ngsocket.cpp:838)         
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
▄  Waiting until the device is joined…  (19.855s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
▀  Waiting until the device is joined…  (24.869s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
 ▀ Waiting until the device is joined…  (29.882s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
 ▀ Waiting until the device is joined…  (34.694s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
 ▄ Waiting until the device is joined…  (39.707s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
▄  Waiting until the device is joined…  (43.116s)WARNING  send multicast to 1 destinations                                                 (/app/core/multicast_layer.cpp:106)  
▀  Waiting until the device is joined…  (44.119s)WARNING  tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT      (/app/core/ngsocket.cpp:838)         
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
▄  Waiting until the device is joined…  (44.72s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)          
 ▀ Waiting until the device is joined…  (45.121s)WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)    
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
▀  Waiting until the device is joined…  (49.733s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
 ▀ Waiting until the device is joined…  (54.748s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
 ▄ Waiting until the device is joined…  (59.761s)WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)         
 ERROR:  timeout                                                                                                                                                                
root@shark-desktop:/ran# WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT      (/app/core/ngsocket.cpp:838)
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT      (/app/core/ngsocket.cpp:838)
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  send multicast to 1 destinations                                                 (/app/core/multicast_layer.cpp:106)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  Sending join request to websetup                                                 (/app/core/websetup.cpp:150)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  tried to serialize unexpected peer to base message type: NAT_INIT_TRANSIENT      (/app/core/ngsocket.cpp:838)
INFO     reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a]              (/app/core/ngsocket.cpp:232)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)
WARNING  peer fc941dc779b8e2fd9df8b003091158fa is blacklisted                             (/app/core/peer_container.cpp:36)

I think the key line is reestablish connection to [fc94:b01d:1803:8dd8:b293:5c7d:7639:932a] it doens´t connect with the websetup server?

The issue is sadly sporadic. On my laptop this works fine and it joins within seconds. Now im trying to set things up on multiple PCs here at the office for a test, but only my laptop was able to register and a collegues laptop after 5-10 mins. Now im working on a desktop and I’ve yet to made it to be able to connect. Also running it at home from my home desktop didnt work for around an hour, and then it suddenly worked without me having done anything special.

This is the husarnet status output:

root@vpn-01:/ran# husarnet status
Version
● CLI:                       2.0.13
● Daemon (running):          2.0.13
● Daemon (binary):           2.0.13

Dashboard URL
● CLI:                       app.husarnet.com
● Daemon:                    app.husarnet.com

Connection status
● Base Server:               188.165.23.196:5582 (UDP)

Readiness
● Is ready to handle data?   yes
● Is ready to join?          yes
● Is joined?                 no

Local
● Husarnet IP:
fc94:59d8:c2b9:c7de:8bab:76c8:a7b5:ba31

Whitelist
fc94:b01d:1803:8dd8:b293:5c7d:7639:932a (websetup)
● active     ● secure         ● tunelled     

A final note, i tried clearing the /var/lib/husarnet folder but that didnt have any effect either.
Also we are, for now, still under the 5 free devices limit as we are still integrating this into our software, so it’s not a client limit either.
if these tests work, we will deploy husarnet on our drones.

Greetings and thank you for your time.

Deniz

Hi @deniz!
Thanks for a wonderful bug report. As per the sporadic websetup connection issues - we will dig deeper into this, but it’ll take more time. As per a temporary test - Husarnet 2.0 added a couple of environment variables to ease a development of integrations like yours. You can change the invocation to HUSARNET_HOSTNAME=vpn-01 HUSARNET_JOIN_CODE=XXXXXXXXXXXXXXXXXXX husarnet-daemon - this should make the daemon attempt a join on it’s own, with no extra scripts and delays required. (Also - you can drop the IPv6 address from join code if you like - the new version of Husrnet supports both short and long versions.) Also - have a look at husarnet daemon wait --help - the joinable option may be helpful to you if you decide not to go with a environment variable route.


Have a good one
Paweł

Dear @pidpawel,

thank you for this information! setting the env variables would definitely make things a bit cleaner.

As for the issue, I think we did hit the 5 nodes limit of the free plan but weren’t aware of it. I am unable now to reproduce it but what happened was that the dashboard’s /network tab showed 1 Network 0 Devices, but when i went to the elements it showed 5 old devices that we didnt use anymore.

A small work-around that did work was manually adding an element to a network. When I did this then i could see from the daemon that the join was successful, but it is unable to join itself the first time. Once i added it manually, it reconnects without a problem.

So i think the joining issue is somehow related to hitting the limit but then me not seeing that reflected in the dasboard/network window. Or it going wrong somewhere in the automatic joining section.

Either way, I am aware of this now. Maybe a return code / warning / when attempting to join a network specifying that the limit was reached, would be useful? Or a dashboard notification that some device attempted to join but was rejected, would help in such a case in the future?

Greetings and thanks again for your time. This product is amazing and it all works seemlessly now. I streamed GPS data from a drone at work to my laptop in a cafe a few days ago and that was super exciting.

Greetings

2 Likes

I’m glad that you’ve found the issue and enoying the product! Adding a proper warning for quota limits is on our todo list for the longest time and one of our requirements for Husarnet 2.1, so it will be done in the next few months. Same goes for making sure that joining an already “enrolled” element/device will add it to proper network automatically.


Have a wonderful day
Paweł