IPv6 timeouts on FreeBSD with PF

After migrating my public-facing cloud servers to shiny new FreeBSD VMs on DigitalOcean, I was somewhat disturbed by the long delays or timeouts I experienced when initiating connections to certain hosts, such as pkg.freebsd.org and update.freebsd.org. This would happen during the DNS lookup phase. The issue was not related to a single DNS server, or even to DNS lookups in general — when I manually did (for example) dig update.freebsd.org @8.8.8.8, I got results immediately. However, when I removed DO’s two default IPv6 nameservers in resolv.conf, problematic lookups sped up, but they were still slow. I had enabled IPv6 when setting up the VMs, and I was using my trusty old pf.conf. Here’s an excerpt:

# (Setting up interfaces, macros, etc)
# ...

block log

pass out all keep state

pass in on $ext_if proto icmp

# (Allowing traffic to certain ports)
# ...

Not having any luck blaming the issue on one specific domain or nameserver, I decided to dig (groan) deeper into the problematic names.

[bethany@host ~]$ dig update.freebsd.org

(...)

;; ANSWER SECTION:
update.freebsd.org.     261     IN      CNAME   update2.freebsd.org.
update2.freebsd.org.    261     IN      A       149.20.53.26

(...)

[bethany@host ~]$ time nc -z 149.20.53.26 80
Connection to 149.20.53.26 80 port [tcp/http] succeeded!

real    0m0.004s
user    0m0.000s
sys     0m0.006s

No issues on IPv4, then. What about IPv6?

[bethany@host ~]$ dig -t AAAA update.freebsd.org

(...)

;; ANSWER SECTION:
update.freebsd.org.     219     IN      CNAME   update2.freebsd.org.
update2.freebsd.org.    12      IN      AAAA    2001:4f8:3:ffe0:406a:0:16:1a

(...)

[bethany@host ~]$ time nc -z 2001:4f8:3:ffe0:406a:0:16:1a 80

real    0m15.632s
user    0m0.000s
sys     0m0.009s

Aha! (Note the lack of a “connection succeeded” message — it timed out.) When I tried to connect to other IPv6 hosts, I got timeouts again and again. What else could there be to check before running off crying to DO support that their image is borked?

[bethany@host ~]$ sudo pfctl -d
No ALTQ support in kernel
ALTQ related functions disabled
pf disabled

[bethany@host ~]$ time nc -z 2001:4f8:3:ffe0:406a:0:16:1a 80
Connection to 2001:4f8:3:ffe0:406a:0:16:1a 80 port [tcp/http] succeeded!

real    0m0.013s
user    0m0.000s
sys     0m0.010s

[bethany@host ~]$ sudo pfctl -e
No ALTQ support in kernel
ALTQ related functions disabled
pf enabled

This was frustrating and suspicious. PF should be keeping TCP state, and I haven’t restricted outbound connections. What else could PF be blocking and causing trouble with? To find out, I checked pflog and initiated another forlorn connection:

[bethany@host ~]$ sudo service pflog onestart
Starting pflog.

[bethany@host ~]$ sudo tcpdump -nettti pflog0
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes
capability mode sandbox enabled


00:00:00.000000 rule 0..16777216/0(match): block in on vtnet0: 2604:my:pre:fix::1 > 2604:my:pre:fix::my:vm: ICMP6, neighbor advertisement, tgt is 2604:my:pre:fix::1, length 32
00:00:01.028675 rule 0..16777216/0(match): block in on vtnet0: 2604:my:pre:fix::1 > 2604:my:pre:fix::my:vm: ICMP6, neighbor advertisement, tgt is 2604:my:pre:fix::1, length 32
00:00:01.002990 rule 0..16777216/0(match): block in on vtnet0: 2604:my:pre:fix::1 > 2604:my:pre:fix::my:vm: ICMP6, neighbor advertisement, tgt is 2604:my:pre:fix::1, length 32

PF had been blocking IPv6’s neighbor discovery protocols (which use ICMPv6)! After making a one-line fix to pf.conf…

pass in on $ext_if proto icmp6

…my issues subsided.

[bethany@host ~]$ time sudo pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.

real    0m0.590s
user    0m0.017s
sys     0m0.009s

Packet Disarray

Despatches from longb4

IPv6 Timeouts on FreeBSD With PF