Lise Ringvold Kristoffersen one of the internal only domains I use that I was attempting DNS validation on is capwap.crc.id.au

As further to this, if I run the same commandset against the LetsEncrypt directory endpoint, things work straight away and a cert is issued.

However, as updating a cert requires this device to reload and dump all access points, I'd rather set it up to use a longer-lived cert so the wifi network doesn't go down as often.

mkon Hi mate,

Still no luck... The output from certbot:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/capwap.crc.id.au.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for capwap.crc.id.au
Hook '--manual-auth-hook' for capwap.crc.id.au ran with output:
Domain: capwap.crc.id.au
DNS update: update add _acme-challenge.capwap.crc.id.au 90 TXT "ywxlRlFxvzqdM3k72ggGU5JLtXTEM3eDXSQ6POBiflE"
Sending to DNS server...
Waiting 5 seconds...
Record should be set, returning to Certbot
Hook '--manual-cleanup-hook' for capwap.crc.id.au ran with output:
Domain: capwap.crc.id.au
DNS update: update delete ${HOST}.${CERTBOT_DOMAIN} TXT
Sending to DNS server...
Failed to renew certificate capwap.crc.id.au with error: All authorizations were not finalized by the CA.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/capwap.crc.id.au/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

The response in the debug log seems to be along the lines of:

2024-01-03 23:16:58,933:DEBUG:urllib3.connectionpool:https://api.buypass.com:443 "POST /acme-v02/authz/4tE1Frv3IhiNJwewYu-WOlxdIQMl1SBxdCXB2ucr-F8 HTTP/1.1" 200 457
2024-01-03 23:16:58,935:DEBUG:acme.client:Received response:
HTTP 200
Content-Encoding: gzip
Content-Type: application/json
Date: Wed, 03 Jan 2024 12:16:58 GMT
Link: <https://api.buypass.com/acme/directory>; rel="index"
Mdc-Correlationid: 519950e5-2059-4e3e-a016-b26811a15685
Replay-Nonce: MWMxNTU1OGMtNjE4NS00MzE2LWI1NGItM2Q5ODQwZDNlZGZk
Vary: Accept-Encoding
Content-Length: 457
Strict-Transport-Security: max-age=63072000

{"identifier":{"type":"dns","value":"capwap.crc.id.au"},"status":"pending","challenges":[{"type":"http-01","url":"https://api.buypass.com/acme-v02/authz/4tE1Frv3IhiNJwewYu-WOlxdIQMl1SBxdCXB2ucr-F8/1","status":"pending","token":"F9F89119
A401228CC29BD020547EB5DBF5CF8951"},{"type":"dns-01","url":"https://api.buypass.com/acme-v02/authz/4tE1Frv3IhiNJwewYu-WOlxdIQMl1SBxdCXB2ucr-F8/2","status":"processing","validated":"2024-01-03T12:15:18Z","token":"715BEF9EF20817246E81150E9
D1E80625F84E0F8","error":{"type":"compound","detail":"Errors during validation","subproblems":[{"type":"urn:ietf:params:acme:error:incorrectResponse","detail":"Response received didn't match the challenge's requirements","code":0},{"typ
e":"urn:ietf:params:acme:error:incorrectResponse","detail":"Response received didn't match the challenge's requirements","code":0},{"type":"urn:ietf:params:acme:error:incorrectResponse","detail":"Response received didn't match the chall
enge's requirements","code":0},{"type":"urn:ietf:params:acme:error:incorrectResponse","detail":"Response received didn't match the challenge's requirements","code":0},{"type":"urn:ietf:params:acme:error:incorrectResponse","detail":"Resp
onse received didn't match the challenge's requirements","code":0},{"type":"urn:ietf:params:acme:error:incorrectResponse","detail":"Response received didn't match the challenge's requirements","code":0},{"type":"urn:ietf:params:acme:err
or:incorrectResponse","detail":"Response received didn't match the challenge's requirements","code":0},{"type":"urn:ietf:params:acme:error:incorrectResponse","detail":"Response received didn't match the challenge's requirements","code":
0},{"type":"urn:ietf:params:acme:error:incorrectResponse","detail":"Response received didn't match the challenge's requirements","code":0}],"code":0}}],"wildcard":false}
2024-01-03 23:16:58,935:DEBUG:acme.client:Storing nonce: MWMxNTU1OGMtNjE4NS00MzE2LWI1NGItM2Q5ODQwZDNlZGZk
2024-01-03 23:16:58,939:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
File "/usr/lib/python3.12/site-packages/certbot/_internal/auth_handler.py", line 216, in _poll_authorizations
raise errors.AuthorizationError('All authorizations were not finalized by the CA.')
certbot.errors.AuthorizationError: All authorizations were not finalized by the CA.

mkon That's strange... I altered the script to not return to certbot until it had a positive response from both 8.8.8.8 and 1.1.1.1 - and get the following:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for capwap.crc.id.au
Hook '--manual-auth-hook' for capwap.crc.id.au ran with output:
Thu Jan  4 01:06:48 AM AEDT 2024
Domain: capwap.crc.id.au
DNS update: update add _acme-challenge.capwap.crc.id.au 30 TXT "ywxlRlFxvzqdM3k72ggGU5JLtXTEM3eDXSQ6POBiflE"
Sending to DNS server...
Validating TXT vs 8.8.8.8 and 1.1.1.1
Output from 8.8.8.8:
_acme-challenge.capwap.crc.id.au. 30 IN       TXT     "ywxlRlFxvzqdM3k72ggGU5JLtXTEM3eDXSQ6POBiflE"
_acme-challenge.capwap.crc.id.au. 30 IN        RRSIG   TXT 13 5 30 20240117060122 20240103130648 16016 crc.id.au. UhRc1YOd36l/Xd4phv+paIEC/uSgZGrjTyoOJe3jjRGhzbDwy/fg+ywd e2ub7uINZ9zkN6LVEVAZvlkRGvKXGw==
Output from 1.1.1.1:
_acme-challenge.capwap.crc.id.au. 30 IN       TXT     "ywxlRlFxvzqdM3k72ggGU5JLtXTEM3eDXSQ6POBiflE"
_acme-challenge.capwap.crc.id.au. 30 IN        RRSIG   TXT 13 5 30 20240117060122 20240103130648 16016 crc.id.au. UhRc1YOd36l/Xd4phv+paIEC/uSgZGrjTyoOJe3jjRGhzbDwy/fg+ywd e2ub7uINZ9zkN6LVEVAZvlkRGvKXGw==

Yet the validation still fails.

Is it possible that there's a DNSSEC error somewhere in the chain?

Given this thread with a similar problem that does NOT have DNSSEC enabled, there goes that theory.

https://community.buypass.com/t/35y3jpb/problem-validating-certain-subdomains

mkon Do you have any other suggestions for troubleshooting? As if all I change is the directory server in certbot to LetsEncrypt, the certificates are validated instantly.

Well, I just set up to do a packet capture on all 3 DNS servers - ran the cert generation stuff once, saw nothing on any of the nameservers.

Checked my command line on the packet dump, ran it again and the certs were issued successfully and I saw 2 x NS servers get the verification query.

Not exactly sure what was going on, but I now have a valid cert...

  mkon Annoyingly, while it worked for that domain, I'm trying another - and hitting the same problem - even though its using the same nameservers, certbot instance etc etc :(

I'm wondering if I'm hitting a rate limit - being as I've renewed quite a few certs on my domain in the past few days?

I'm doing a packet dump on all 3 primary DNS servers, and none are getting the request for the _acme-challenge TXT record.

mkon Thanks for checking... The one successful validation I got via DNS, the probe came from 172.253.x.x - and it hit ns1.crc.id.au.

When these domains fail, I see nothing hitting ns1, ns2 or ns3.

I don't know if you have access or not, but is it possible to attempt to resolve stuff from these resolvers?

Interestingly, I just tried repeating the same command as before, and now the second one I hit this issue with is now working...

I'm not exactly sure what is going on, but it seems to be at the actual DNS TXT record resolution phase.

Buypass Customer Support I did manage to get the certs eventually.

A couple of the services are not mission critical - so I can try a force renew of those certs if that would be helpful?