Mosquitto doesn't start (can't open pwfile) #67

timocarnill · 2024-11-21T17:22:27Z

When I run quicksetup, it all seems to run well, but at the last part mosquitto won't start, seemingly because it's not able to open a password file. (my hostname is wireguard-vpn because that's what I used to use this VPS for, but wireguard is removed now. I do have tailscale installed and running if that's relevant)

/var/log/mosquitto/mosquitto.log

1732208292: mosquitto version 2.0.11 starting
1732208292: Config loaded from /etc/mosquitto/mosquitto.conf.
1732208292: Starting in local only mode. Connections will only be possible from clients running on this machine.
1732208292: Create a configuration file which defines a listener to allow remote access.
1732208292: For more details see https://mosquitto.org/documentation/authentication-methods/
1732208292: Opening ipv4 listen socket on port 1883.
1732208292: Opening ipv6 listen socket on port 1883.
1732208292: mosquitto version 2.0.11 running
1732208314: mosquitto version 2.0.11 terminating
1732208314: Saving in-memory database to /var/lib/mosquitto//mosquitto.db.
1732208314: mosquitto version 2.0.11 starting
1732208314: Config loaded from /etc/mosquitto/mosquitto.conf.
1732208314: Starting in local only mode. Connections will only be possible from clients running on this machine.
1732208314: Create a configuration file which defines a listener to allow remote access.
1732208314: For more details see https://mosquitto.org/documentation/authentication-methods/
1732208314: Opening ipv4 listen socket on port 1883.
1732208314: Opening ipv6 listen socket on port 1883.
1732208314: mosquitto version 2.0.11 running
1732208323: New connection from 127.0.0.1:56116 on port 1883.
1732208323: New client connected from 127.0.0.1:56116 as ot-recorder-wireguard-vpn-12797 (p2, c0, k60).
1732208333: mosquitto version 2.0.11 terminating
1732208333: Saving in-memory database to /var/lib/mosquitto//mosquitto.db.
1732208333: mosquitto version 2.0.11 starting
1732208333: Config loaded from /etc/mosquitto/mosquitto.conf.
1732208333: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732208333: Error opening password file "/etc/mosquitto/mosquitto.pw".

The last 4 lines repeat a few times

Output of doas systemctl status mosquitto.service

× mosquitto.service - Mosquitto MQTT Broker
     Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-11-21 18:10:14 CET; 6min ago
   Duration: 18.858s
       Docs: man:mosquitto.conf(5)
             man:mosquitto(8)
    Process: 19889 ExecStartPre=/bin/mkdir -m 740 -p /var/log/mosquitto (code=exited, status=0/SUCCESS)
    Process: 19890 ExecStartPre=/bin/chown mosquitto /var/log/mosquitto (code=exited, status=0/SUCCESS)
    Process: 19891 ExecStartPre=/bin/mkdir -m 740 -p /run/mosquitto (code=exited, status=0/SUCCESS)
    Process: 19892 ExecStartPre=/bin/chown mosquitto /run/mosquitto (code=exited, status=0/SUCCESS)
    Process: 19893 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=exited, status=13)
   Main PID: 19893 (code=exited, status=13)
        CPU: 10ms

Nov 21 18:10:14 wireguard-vpn systemd[1]: mosquitto.service: Main process exited, code=exited, status=13/n/a
Nov 21 18:10:14 wireguard-vpn systemd[1]: mosquitto.service: Failed with result 'exit-code'.
Nov 21 18:10:14 wireguard-vpn systemd[1]: Failed to start mosquitto.service - Mosquitto MQTT Broker.
Nov 21 18:10:14 wireguard-vpn systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
Nov 21 18:10:14 wireguard-vpn systemd[1]: Stopped mosquitto.service - Mosquitto MQTT Broker.
Nov 21 18:10:14 wireguard-vpn systemd[1]: mosquitto.service: Start request repeated too quickly.
Nov 21 18:10:14 wireguard-vpn systemd[1]: mosquitto.service: Failed with result 'exit-code'.
Nov 21 18:10:14 wireguard-vpn systemd[1]: Failed to start mosquitto.service - Mosquitto MQTT Broker.

Output of doas journalctl -xeu mosquitto.service

░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mosquitto.service has finished with a failure.
░░ 
░░ The job identifier is 2199 and the job result is failed.
Nov 21 18:10:14 wireguard-vpn systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ Automatic restarting of the unit mosquitto.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Nov 21 18:10:14 wireguard-vpn systemd[1]: Stopped mosquitto.service - Mosquitto MQTT Broker.
░░ Subject: A stop job for unit mosquitto.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit mosquitto.service has finished.
░░ 
░░ The job identifier is 2264 and the job result is done.
Nov 21 18:10:14 wireguard-vpn systemd[1]: mosquitto.service: Start request repeated too quickly.
Nov 21 18:10:14 wireguard-vpn systemd[1]: mosquitto.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit mosquitto.service has entered the 'failed' state with result 'exit-code'.
Nov 21 18:10:14 wireguard-vpn systemd[1]: Failed to start mosquitto.service - Mosquitto MQTT Broker.
░░ Subject: A start job for unit mosquitto.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mosquitto.service has finished with a failure.
░░ 
░░ The job identifier is 2264 and the job result is failed.

my configuration.yaml: (sensitive information replaced with #)

# configuration for OwnTracks backend quicksetup

# Specify your DNS domain, the name by which this OwnTracks
# backend will be reacheable. We use this to enroll in
# Let's Encrypt certificates for this domain on your behalf
#
# Make sure you replace 'owntracks.example.net' by the correct
# name for your installation.
#

dns_domain: "#"

# Specify your email address. We use this when signing up on your
# behalf to Let's Encrypt and for nothing else. If you don't
# configure an email address, we set up without SSL/TLS

email: "#"

# We strongly recommend you sign up for the free reverse geo
# service at OpenCage. It costs you nothing, and they provide
# you with an API key you add here.
#

opencage_apikey: "#"

# Configure yourself and a list of friends who will be using
# this backend. Each friend will use their "username" to login
# to the site and to the apps. Their device is a "devicename"
# (this can be any suitable name, but we recommend you keep it
# generic so as to not have to change this if you move to a
# distinct device; i.e. "myphone" instead of "siemens-e2"),
# and their "tid" is a tracker- id which is used to label users
# on the map. A tid is a string of length two characters.
# Optionally a "password" may be set below; if omitted, passwords
# are generated and stored at /usr/local/owntracks/userdata/*.pass
# on this system.
#
# username and devicename may contain digits and lowercase letters.

friends:
  - { tid: "TC", username: "#",      devicename: "telefoontc" }
  - { tid: "MJ", username: "#",     devicename: "telefoonmj" }
  - { tid: "SL", username: "#",      devicename: "telefoonsl" }
  - { tid: "JF", username: "#",      devicename: "telefoonjf" }
  - { tid: "SB", username: "#",      devicename: "telefoonsb" }
  - { tid: "SG", username: "#",      devicename: "telefoonsg" }
  - { tid: "EC", username: "#",      devicename: "telefoonec" }
  - { tid: "LN", username: "#",      devicename: "telefoonln" }
  - { tid: "VR", username: "#",      devicename: "telefoonvr" }


# ---- advanced options below

# Path to an optional Lua script the Recorder should load at
# startup.

# lua_script: "/path/to/example.lua"

The text was updated successfully, but these errors were encountered:

jpmens · 2024-11-21T17:36:01Z

/etc/mosquitto/mosquitto.pw is a file which is installed during ./bootstrap.sh, and it greatly surprises me that the file shouldn't be there.

Please run ./bootstrap.sh again and look for the task named

mosquitto: build mosquitto.pw password file from user passwords

Please show us the output of that (it ought to be yellow or green).

After bootrap, please check with ls -l /etc/mosquitto/ whether you see the file and if so, show us its permissions please.

timocarnill · 2024-11-21T18:15:03Z

The file is there. It has r--r----- permissions and is owned by root.

TASK [mosquitto: build mosquitto.pw password file from user passwords] **************************************************************************************************************************
changed: [localhost]

ls -la /etc/mosquitto

total 44
drwxr-xr-x  5 root root 4096 Nov 21 19:13 .
drwxr-xr-x 88 root root 4096 Nov 21 17:58 ..
-rw-r--r--  1 root root  230 Jun  9  2021 aclfile.example
drwxr-xr-x  2 root root 4096 Nov 21 17:58 ca_certificates
drwxr-xr-x  2 root root 4096 Nov 21 17:58 certs
drwxr-xr-x  2 root root 4096 Nov 21 17:58 conf.d
-r--r-----  1 root root 1652 Nov 21 17:58 mosquitto.acl
-rw-r--r--  1 root root  354 Sep 30  2023 mosquitto.conf
-r--r-----  1 root root 1321 Nov 21 19:13 mosquitto.pw
-rw-r--r--  1 root root   23 Jun  9  2021 pskfile.example
-rw-r--r--  1 root root  355 Jun  9  2021 pwfile.example

jpmens · 2024-11-21T20:20:36Z

The file belongs to user root and Mosquitto seems to be launching as user mosquitto and it, it is true, cannot open that file (nor mosquitto.acl).

Would you please, as a quick fix, change their ownership:

doas chown mosquitto: /etc/mosquitto/mosquitto.{acl,pw}

(The colon behind the username is no mistake; the group will automatically be chosen from the user's /etc/passwd entry.)

You should then be able to launch mosquitto:

doas systemctl start mosquitto

Should that be the fix, we'll fix permanently in our setup obviously. I think Mosquitto is tightening down on their security.

Please also then show content of /etc/os-release and output of dpkg -l | grep mosquitto.

timocarnill · 2024-11-21T22:08:39Z

Also doesn't work. doas systemctl start mosquitto gives

Job for mosquitto.service failed because the control process exited with error code.
See "systemctl status mosquitto.service" and "journalctl -xeu mosquitto.service" for details.

Even after giving those 2 files "full" permissions (so rwxrwxrwx). The same error message is given with systemctl restart instead of systemctl start. After rerunning bootstrap.sh the permissions get reset.

doas systemctl status mosquitto.service gives:

× mosquitto.service - Mosquitto MQTT Broker
     Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-11-21 22:57:24 CET; 4min 4s ago
       Docs: man:mosquitto.conf(5)
             man:mosquitto(8)
    Process: 7178 ExecStartPre=/bin/mkdir -m 740 -p /var/log/mosquitto (code=exited, status=0/SUCCESS)
    Process: 7179 ExecStartPre=/bin/chown mosquitto /var/log/mosquitto (code=exited, status=0/SUCCESS)
    Process: 7180 ExecStartPre=/bin/mkdir -m 740 -p /run/mosquitto (code=exited, status=0/SUCCESS)
    Process: 7181 ExecStartPre=/bin/chown mosquitto /run/mosquitto (code=exited, status=0/SUCCESS)
    Process: 7182 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf (code=exited, status=3)
   Main PID: 7182 (code=exited, status=3)
        CPU: 10ms

Nov 21 22:57:24 wireguard-vpn systemd[1]: mosquitto.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Nov 21 22:57:24 wireguard-vpn systemd[1]: mosquitto.service: Failed with result 'exit-code'.
Nov 21 22:57:24 wireguard-vpn systemd[1]: Failed to start mosquitto.service - Mosquitto MQTT Broker.
Nov 21 22:57:24 wireguard-vpn systemd[1]: mosquitto.service: Scheduled restart job, restart counter is at 5.
Nov 21 22:57:24 wireguard-vpn systemd[1]: Stopped mosquitto.service - Mosquitto MQTT Broker.
Nov 21 22:57:24 wireguard-vpn systemd[1]: mosquitto.service: Start request repeated too quickly.
Nov 21 22:57:24 wireguard-vpn systemd[1]: mosquitto.service: Failed with result 'exit-code'.
Nov 21 22:57:24 wireguard-vpn systemd[1]: Failed to start mosquitto.service - Mosquitto MQTT Broker.
Nov 21 22:58:30 wireguard-vpn systemd[1]: mosquitto.service: Unit cannot be reloaded because it is inactive.

The only difference I could see; "code=exited, status=3/NOTIMPLEMENTED". No idea what that means though... There didn't seem to be a meaningful difference in /var/log/mosquitto/mosquitto.log, just the same 4 lines repeating at the end again.

Content of /etc/os-release:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Output of dpkg -l | grep mosquitto:

ii  libmosquitto1:amd64             2.0.11-1.2+deb12u1               amd64        MQTT version 5.0/3.1.1/3.1 client library
ii  mosquitto                       2.0.11-1.2+deb12u1               amd64        MQTT version 5.0/3.1.1/3.1 compatible message broker
ii  mosquitto-clients               2.0.11-1.2+deb12u1               amd64        Mosquitto command line MQTT clients

jpmens · 2024-11-22T07:24:30Z

Please try restarting mosquito as earlier and show the errors in the last 20 or so lines of /var/log/mosquitto/mosquitto.log

jpmens · 2024-11-22T08:33:28Z

@timocarnill a side note if I may: please note that giving files permissions 0777 (rwxrwxrwx) is almost always wrong!

jpmens · 2024-11-22T08:35:16Z

Sorry @timocarnill and @amrul-jamrul are having trouble. I'm quite sure it's a permission / ownership problem. If either of you could please, shortly after attempting a Mosquitto start show the last lines of the mosquitto log, I'm sure we'll find it: (Something must have recently changed)

$ sudo systemctl restart mosquitto
$ sudo tail -30 /var/log/mosquitto/mosquitto.log

addresses #67

jpmens · 2024-11-22T08:47:11Z

I think it's now solved, and apologies for the error which I introduced (or rather: didn't properly fix) in #66

If you would both please:

$ git pull
$ sudo ./bootstrap.sh

the ownership / permissions of mosquitto.{acl,pw} should be correctly set and result in

$ ls -l /etc/mosquitto/
drwxr-xr-x 2 root      root 4096 Mar 18  2024 ca_certificates
drwxr-xr-x 3 root      root 4096 Jun  3 12:19 certs
drwxr-xr-x 2 root      root 4096 Jul 26 16:02 conf.d
-r--r----- 1 mosquitto mosquitto 1025 Jun 16 10:22 mosquitto.acl
-rw-r--r-- 1 root      root       354 Sep 30  2023 mosquitto.conf
-r--r----- 1 mosquitto mosquitto  831 Nov 22 09:49 mosquitto.pw

$ ls -l /etc/mosquitto/conf.d/owntracks.conf
-r--r----- 1 mosquitto mosquitto 818 Jul 26 16:02 /etc/mosquitto/conf.d/owntracks.conf

amrul-jamrul · 2024-11-22T23:20:56Z

The setup script works for me now, thank you.

timocarnill · 2024-11-23T16:33:00Z

Still not working for me after git pull and rerunning bootstrap.sh. Same error.
How would I go about completely removing everything from quicksetup? Maybe there's some kind of remnant of the old installation?

jpmens · 2024-11-23T16:52:13Z

Sorry to hear you're still having trouble. I do hope you ran the git pull from within the directory for the quicksetup you originally cloned. (Note that if you downloaded a zip or tarball, then that won't work.)

You can double-check the pull worked:

$ git log | head -3
commit b56697ab022564bb8949a9dc709d6498529ab828
Author: Jan-Piet Mens <[email protected]>
Date:   Fri Nov 22 09:50:38 2024 +0100

To answer your question: the issue being Mosquitto, we can wipe that installation with these commands:

$ sudo apt remove mosquitto
$ sudo rm -r /etc/mosquitto/

The former will uninstall the package, and the latter will clear the directory completely.

Re-run sudo ./bootstrap.sh to get both the package installed and the configuration directory populated. After this you ought to see the files belonging to user mosquitto / group mosquitto as illustrated above.

jpmens · 2024-11-23T16:53:10Z

Should that still not work, which would surprise me a bit, please provide the tail 30 lines of the mosquitto.log for us to see.

timocarnill · 2024-11-23T20:27:57Z

I'm at a loss... Still doesn't work.
I removed mosquitto, removed its directory, git pulled in the right directory...

output of ls -l /etc/mosquitto

total 20
drwxr-xr-x 2 root      root      4096 Nov 23 21:18 ca_certificates
drwxr-xr-x 2 root      root      4096 Nov 23 21:18 certs
drwxr-xr-x 2 root      root      4096 Nov 23 21:18 conf.d
-r--r----- 1 mosquitto mosquitto 1652 Nov 23 21:18 mosquitto.acl
-r--r----- 1 mosquitto mosquitto 1321 Nov 23 21:18 mosquitto.pw

sudo tail -n 30 /var/log/mosquitto/mosquitto.log

1732210094: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732210094: Error opening password file "/etc/mosquitto/mosquitto.pw".
1732210094: mosquitto version 2.0.11 starting
1732210094: Config loaded from /etc/mosquitto/mosquitto.conf.
1732210094: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732210094: Error opening password file "/etc/mosquitto/mosquitto.pw".
1732210094: mosquitto version 2.0.11 starting
1732210094: Config loaded from /etc/mosquitto/mosquitto.conf.
1732210094: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732210094: Error opening password file "/etc/mosquitto/mosquitto.pw".
1732212813: mosquitto version 2.0.11 starting
1732212813: Config loaded from /etc/mosquitto/mosquitto.conf.
1732212813: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732212813: Error opening password file "/etc/mosquitto/mosquitto.pw".
1732212814: mosquitto version 2.0.11 starting
1732212814: Config loaded from /etc/mosquitto/mosquitto.conf.
1732212814: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732212814: Error opening password file "/etc/mosquitto/mosquitto.pw".
1732212814: mosquitto version 2.0.11 starting
1732212814: Config loaded from /etc/mosquitto/mosquitto.conf.
1732212814: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732212814: Error opening password file "/etc/mosquitto/mosquitto.pw".
1732212814: mosquitto version 2.0.11 starting
1732212814: Config loaded from /etc/mosquitto/mosquitto.conf.
1732212814: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732212814: Error opening password file "/etc/mosquitto/mosquitto.pw".
1732212814: mosquitto version 2.0.11 starting
1732212814: Config loaded from /etc/mosquitto/mosquitto.conf.
1732212814: Error: Unable to open pwfile "/etc/mosquitto/mosquitto.pw".
1732212814: Error opening password file "/etc/mosquitto/mosquitto.pw".

jpmens · 2024-11-23T21:01:17Z

Is that the complete output of ls -l /etc/mosquitto ?

jpmens · 2024-11-23T21:02:34Z

please show dppk -l | grep mosquitto output.

timocarnill changed the title ~~Mosquitto can't open pwfile~~ Mosquitto doesn't start (can't open pwfile) Nov 21, 2024

amrul-jamrul mentioned this issue Nov 22, 2024

MQTT Quickstart stuck at Mosquitto Service owntracks/recorder#508

Closed

jpmens added a commit that referenced this issue Nov 22, 2024

ensure correct ownership of mosquitto.{acl,pw}

a3c65a1

addresses #67

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mosquitto doesn't start (can't open pwfile) #67

Mosquitto doesn't start (can't open pwfile) #67

timocarnill commented Nov 21, 2024 •

edited

Loading

jpmens commented Nov 21, 2024

timocarnill commented Nov 21, 2024 •

edited

Loading

jpmens commented Nov 21, 2024

timocarnill commented Nov 21, 2024 •

edited

Loading

jpmens commented Nov 22, 2024 via email

jpmens commented Nov 22, 2024

jpmens commented Nov 22, 2024 •

edited

Loading

jpmens commented Nov 22, 2024 •

edited

Loading

amrul-jamrul commented Nov 22, 2024

timocarnill commented Nov 23, 2024

jpmens commented Nov 23, 2024

jpmens commented Nov 23, 2024

timocarnill commented Nov 23, 2024

jpmens commented Nov 23, 2024

jpmens commented Nov 23, 2024

Mosquitto doesn't start (can't open pwfile) #67

Mosquitto doesn't start (can't open pwfile) #67

Comments

timocarnill commented Nov 21, 2024 • edited Loading

jpmens commented Nov 21, 2024

timocarnill commented Nov 21, 2024 • edited Loading

jpmens commented Nov 21, 2024

timocarnill commented Nov 21, 2024 • edited Loading

jpmens commented Nov 22, 2024 via email

jpmens commented Nov 22, 2024

jpmens commented Nov 22, 2024 • edited Loading

jpmens commented Nov 22, 2024 • edited Loading

amrul-jamrul commented Nov 22, 2024

timocarnill commented Nov 23, 2024

jpmens commented Nov 23, 2024

jpmens commented Nov 23, 2024

timocarnill commented Nov 23, 2024

jpmens commented Nov 23, 2024

jpmens commented Nov 23, 2024

timocarnill commented Nov 21, 2024 •

edited

Loading

timocarnill commented Nov 21, 2024 •

edited

Loading

timocarnill commented Nov 21, 2024 •

edited

Loading

jpmens commented Nov 22, 2024 •

edited

Loading

jpmens commented Nov 22, 2024 •

edited

Loading