Husarnet stopped working in Raspberry

[albertoezquerro]

Hello there!

This is Alberto, from The Construct.

I’ve been working during the last few days preparing a demo that involves connecting from a rosject (computer in the cloud) to a Raspberry using Husarnet. I’ve had no problem so far until today. Today, I tried to reproduce the demo again and Husarnet is not working anymore in the Raspberry. When I try to run a husarnet status command, the console just hangs here forever:

theconstruct@rpi4-11:~$ Raspberry Pi 4 Robot>> husarnet status
 INFO:  Daemon does not seem to be running

Do you want to restart Husarnet Daemon now? [y/N]: Yes

 WARNING:  Wasn't able to restart Husarnet Daemon. Try restarting the service manually.
  Waiting until we can communicate with husarnet daemon… (7.833s)
  Waiting until we can communicate with husarnet daemon… (8.034s)
  Waiting until we can communicate with husarnet daemon… (8.235s)
  Waiting until we can communicate with husarnet daemon… (8.636s)

I’ve tried reinstalling Husarnet among other things, but nothing is working. I noticed that when reinstalling Husarnet, I got an exit code of 1:

The following NEW packages will be installed:
  husarnet
0 upgraded, 1 newly installed, 0 to remove and 130 not upgraded.
Need to get 0 B/6721 kB of archives.
After this operation, 13.1 MB of additional disk space will be used.
Selecting previously unselected package husarnet.
(Reading database ... 167448 files and directories currently installed.)
Preparing to unpack .../husarnet_1%3a2.0.180_arm64.deb ...
Unpacking husarnet (1:2.0.180) ...
Setting up husarnet (1:2.0.180) ...
exit status 1
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.


Job for husarnet.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status husarnet.service" and "journalctl -xeu husarnet.service" for details.

I’ve also tried running the suggested commands in the last line, and this is the output obtained:

theconstruct@rpi4-11:~$ Raspberry Pi 4 Robot>> systemctl status husarnet.service
● husarnet.service - Husarnet
     Loaded: loaded (/etc/systemd/system/husarnet.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: core-dump) since Tue 2024-07-02 18:29:02 CEST; 2s ago
    Process: 5707 ExecStart=/usr/bin/husarnet-daemon (code=dumped, signal=ABRT)
   Main PID: 5707 (code=dumped, signal=ABRT)
        CPU: 7ms

For the journalctl command:

02 18:29:19 rpi4-11 systemd[1]: Starting Husarnet...
░░ Subject: A start job for unit husarnet.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit husarnet.service has begun execution.
░░
░░ The job identifier is 34420.
Jul 02 18:29:19 rpi4-11 husarnet-daemon[5725]: libc++abi: terminating with uncaught exception of type nlohmann::detail::parse_error: [json.exception.par>
Jul 02 18:29:19 rpi4-11 systemd[1]: husarnet.service: Main process exited, code=dumped, status=6/ABRT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit husarnet.service has exited.
░░
░░ The process' exit code is 'dumped' and its exit status is 6.
Jul 02 18:29:19 rpi4-11 systemd[1]: husarnet.service: Failed with result 'core-dump'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit husarnet.service has entered the 'failed' state with result 'core-dump'.
Jul 02 18:29:19 rpi4-11 systemd[1]: Failed to start Husarnet.
░░ Subject: A start job for unit husarnet.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit husarnet.service has finished with a failure.
░░
░░ The job identifier is 34420 and the job result is failed.

Any ideas of what could be wrong? Thanks in advance for the help.

Best,

[Filip-Husarnet]

Hi Alberto

Can you send us output from following command?

sudo journalctl --unit husarnet --since yesterday

Please send this output either to me via private message or send us mail to support@husarnet.com. (If you decide to send us mail please mention this thread in it.)

Regards, Filip

[albertoezquerro]

Just sent the e-mail. Thanks again.

[pidpawel]

Thanks for sending the logs, unfortunately they do not specify the origin of the problem (I’ll add marking that to our todolist later). Fortunately though, there are only 3 places where we parse JSON in daemon.

1. license - one of the very first things daemon does is parsing the license file (local one if it was already downloaded, downloading one if not)
2. config storage - some high level variables are stored on disk in a config file
3. local API server accepts JSON messages as input

I doubt it’s the last one in this case, as it seems to have a problem very early in the process. As per the first two - I’d suggest removing /var/lib/husarnet/license.json first (it’s a safe operation, daemon should re-download it) and if that not helps - removing /var/lib/husarnet/config.json too (this would require re-joining the device, but would not change it’s IPv6 address).

It’d be super helpful if you sent us contents (same address as before) of those files before removing them - usually it’s some case of network issue/adblocker/temporary service error which mangles license file in the process, or a case where there was no space left on the device and either of these files gets overwritten with an empty file while trying to update it. Based on the log in the email - I’d suspect one of those files is empty.

–
Have a wonderful day
Paweł

[albertoezquerro]

Removing the license.json file didn’t solve the problem, but removing config.json did solve it! I’ve sent both files via e-mail as requested. Many thanks for the help!

Best,