title: "Just Another Measurement of Extension header Survivability (JAMES)" abbrev: "JAMES" category: info submissiontype: IETF docname: draft-vyncke-v6ops-james-latest ipr: trust200902 area: "Operations and Management" workgroup: "IPv6 Operations" keyword: Internet-Draft venue: group: "IPv6 Operations" type: "Working Group" mail: "[email protected]" arch: "https://mailarchive.ietf.org/arch/browse/v6ops/" github: "evyncke/v6ops-james" latest: "https://evyncke.github.io/v6ops-james/draft-vyncke-v6ops-james.html"
stand_alone: yes smart_quotes: no pi: [toc, sortrefs, symrefs]
ins: É. Vyncke
name: Éric Vyncke
organization: Cisco
street: De Kleetlaan 64
code: 1831
city: Diegem
country: Belgium
email: [email protected]
- ins: R. Léas name: Raphaël Léas organization: Université de Liège city: Liège country: Belgium email: [email protected]
- ins: J. Iurman
name: Justin Iurman
organization: Université de Liège
street:
- Institut Montefiore B28
- Allée de la Découverte 10 code: 4000 city: Liège country: Belgium email: [email protected]
normative: IANA_IPV6_PARAMS: title: Internet Protocol Version 6 (IPv6) Parameters, Destination Options and Hop-by-Hop Options target: https://www.iana.org/assignments/ipv6-parameters/ipv6-parameters.xhtml#ipv6-parameters-2
informative: TIER1: title: Tier 1 network target: https://en.wikipedia.org/wiki/Tier_1_network ALEXA: title: The top 500 sites on the web target: https://www.alexa.com/topsites GITHUB: title: james target: https://gitlab.uliege.be/Benoit.Donnet/james author: - name: Raphaël Léas org: Université de Liège
MLAT_PEERING: title: Inferring Multilateral Peering target: https://catalog.caida.org/details/paper/2013_inferring_multilateral_peering/ seriesinfo: DOI: 10.1145/2535372.2535390 date: 2013-12 author: - name: Vasileios Giotsas org: University College London - name: Shi Zhou org: University College London - name: Matthew Luckie org: CAIDA, San Diego Supercomputer Center, University Of California San Diego - name: Kc Claffy org: CAIDA, San Diego Supercomputer Center, University Of California San Diego
--- abstract In 2016, RFC7872 has measured the drop of packets with IPv6 extension headers. This document presents a slightly different methodology with more recent results. It is still work in progress.
--- middle
In 2016, {{?RFC7872}} has measured the drop of packets with IPv6 extension headers on their transit over the global Internet. This document presents a slightly different methodology with more recent results. Since then, {{?RFC9288}} has provided some recommendations for filtering transit traffic, so there may be some changes in providers policies. Also, {{?RFC9098}} raises awareness about the operational and security implications of IPv6 extension headers and present reasons why some networks would drop them intentionally.
It is still work in progress, but the authors wanted to present some results at IETF-113 (March 2022). The code is open source and is available at {{GITHUB}}.
In a first phase, the measurement is done between collaborating IPv6 nodes, a.k.a. vantage points, spread over the Internet and multiple Autonomous Systems (ASs). As seen in {{analysed_as}}, the source/destination/transit ASs include some "tier-1" providers per {{TIER1}}, so, they are probably representative of the global Internet core.
Relying on collaborating nodes has some benefits:
-
propagation can be measured even in the absence of any ICMP message or reply generated by the destination;
-
traffic timing can be measured accurately to answer whether extension headers are slower than plain IPv6 packets;
-
traffic can be captured into .pcap {{?I-D.draft-ietf-opsawg-pcap}} file at the source and at the destination for later analysis.
Future phases will send probes to non-collaborating nodes with a much reduced probing speed. The destination will include {{ALEXA}} top-n websites, popular CDN, as well as random prefix from the IPv6 global routing table. A revision of this IETF draft will describe those experiments.
Several servers were used worldwide. {{table_vantage}} lists all the vantage points together with their AS number and country.
{::include ./vantage_as.inc} {: #table_vantage title="All vantage AS"}
During first phase (traffic among fully-meshed collaborative nodes), {{table_analysed_as}} show the ASs for which our probes have collected data.
{::include ./analysed_as.inc} {: #table_analysed_as title="All AS (source/destination/transit)"}
The table attributes some tier qualification to some ASs based on the Wikipedia page {{TIER1}}, but there is no common way to decide who is a tier-1. Based on some CAIDA research, all the above (except GO6, which is a stub network) are transit providers.
While this document lists some operators, the intent is not to build a wall of fame or a wall of shame but more to get an idea about which kind of providers drop packets with extension headers and how widespread the drop policy is enforced and where, i.e., in the access provider or in the core of the Internet.
Comparing the traceroutes with and without extension headers allows the attribution of a packet drop to one AS. But, this is not an easy task as inter-AS links often use IPv6 address of only one AS (if not using link-local per {{?RFC7404}}). This document uses the following algorithm to attribute the drop to one AS for packet sourced in one AS and then having a path traversing AS#foo just before AS#bar:
-
if the packet drop happens at the first router (i.e., hop limit == 1 does not trigger an ICMP hop-limit exceeded), then the drop is assumed to this AS as it is probably an ingress filter on the first router (i.e., the hosting provider in most of the cases - except if collocated with an IXP).
-
if the packet drop happens in AS#foo after one or more hop(s) in AS#bar, then the drop is assumed to be in AS#foo ingress filter on a router with an interface address in AS#foo
-
if the packet drop happens in AS#bar after one or more hop(s) in AS#bar before going to AS#foo, then the drop is assumed to be in AS#foo ingress filter on a router with an interface address in AS#bar
In several cases, the above algorithm was not possible (e.g., some intermediate routers do not generate an ICMP unreachable hop limit exceeded even in the absence of any extension headers), then the drop is not attributed. Please also note that the goal of this document is not to 'point fingers to operators' but more to evaluate the potential impact. I.e., a tier-1 provider dropping packets with extension headers has a much bigger impact on the Internet traffic than an access provider.
Future revision of this document will use the work of {{MLAT_PEERING}}. Readers are urged not to rely on the AS attribution in this document version.
In the first phase among collaborating vantage points, packets always contained either a UDP payload or a TCP payload, the latter is sent with only the SYN flag set and with data as permitted by section 3.5 of {{!RFC9293}} (2nd paragraph). A usual traceroute is done with only the UDP/TCP payload without any extension header with varying hop-limit in order to learn the traversed routers and ASs. Then, several UDP/TCP probes are sent with a set of extension headers:
-
hop-by-hop options header containing:
- one PadN option for a length of 8 octets
- one unknown option with the "discard" bits for a length of 8 octets
- one unknown option (two sets: with "discard" and "skip" bits) for a length of 256 and 512 octets
-
destination options header containing:
- one PadN option for a length of 8 octets
- one unknown option with the "discard" bits for a length of 8 octets
- one unknown option (two sets: with "discard" and "skip" bits) for a length of 16, 32, 64, 128, 256, and 512 octets
- one unknown option (with "skip" bits) for a length of 24, 40, 48, and 56 octets
-
routing header with routing types from 0 to 6 inclusive
-
fragment header of varying frame length 512, 1280, and 1500 octets:
- atomic fragment (i.e., M-flag = 0 and offset = 0)
- non-atomic first fragment (i.e., M-flag = 1 and offset = 0)
-
encapsulation security payload (ESP) header with dummy SPI followed by UDP/TCP header and a 38 octets payload
-
authentication header (AH) with dummy SPI followed by UDP/TCP header and a 38 octets payload
In the above, length is the length of the extension header itself except for the fragmentation header where the length is the IP packet length (i.e., including the IPv6, and TCP/UDP headers + payload). Also, an unknown option means an option with an unassigned code in the IANA registry {{IANA_IPV6_PARAMS}}.
For hop-by-hop and destination options headers, the choice was made to use one unknown option instead of multiple consecutive PadN options in order to avoid packets from being discarded on the destination. Indeed, the Linux kernel does not accept consecutive Pad1 or PadN options if their total size exceeds 7 octets. Not only multiple PadN options violate section 2.1.9.5 of {{?RFC4942}}, but it is also considered as suspicious (see section 5.3 of {{?BCP220}}). Nevertheless, for comparative purposes, multiple PadN options were used for experiments of length 256 octets. In that very specific case, drops on the destination are not considered as drops.
In addition to the above extension headers, other probes were sent with next header field of IPv6 header set to:
-
59, which is "no next header", especially whether extra octets after the no next header as section 4.7 {{!RFC8200}} requires that "those octets must be ignored and passed on unchanged if the packet is forwarded"
-
143, which is Ethernet payload (see section 10.1 of {{?RFC8986}})
This section presents the current results out of phase 1 (collaborating vantage points) testing. Probe packets were sent between all pairs of vantage points with a hop-limit from 1 to the number of hops between the two vantage points and for all the extension headers described in {{tested_eh}}.
{{table_rh_types}} lists all routing header types and the percentage of experiments that were successful, i.e., packets with routing header reaching their destination, both for UDP and TCP:
| Routing Header Type | UDP | TCP | | 0 | 74.3% | 71.2% | | 1 | 88.3% | 81.4% | | 2 | 97.4% | 90.4% | | 3 | 97.6% | 91.3% | | 4 | 78.8% | 72.6% | | 5 | 97.4% | 90.9% | | 6 | 97.4% | 90.0% | {: #table_rh_types title="Per Routing Header Types Transmission"}
{{table_drop_rh0}} and {{table_drop_rh1}} respectively list ASs that drop packets with the routing header type 0 (the original source routing header, which is now deprecated) and packets with the routing header type 1 (NIMROD {{?RFC1753}}, which is now deprecated).
{::include ./drop_rh0_as.inc} {: #table_drop_rh0 title="ASs dropping Routing Header Type 0"}
{::include ./drop_rh1_as.inc} {: #table_drop_rh1 title="ASs dropping Routing Header Type 1"}
Regarding the routing type 0, it is possibly due to a strict implementation of {{?RFC5095}} but it is expected that no packet with such routing type would be transmitted anymore. So, this is not surprising. The same reasoning could be applied to the routing type 1.
{{table_drop_rh4}} lists ASs that drop packets with the routing header type 4 (Segment Routing Header {{?RFC8754}}).
{::include ./drop_rh4_as.inc} {: #table_drop_rh4 title="ASs dropping Routing Header Type 4"}
This drop of SRH was to be expected as SRv6 is specified to run only in a limited domain.
Other routing header types (2 == mobile IPv6 {{?RFC6275}}, 3 == RPL {{?RFC6554}}, and even 5 == CRH-16 and 6 == CRH-32{{?I-D.draft-bonica-6man-comp-rtg-hdr}}) can be transmitted over the global Internet without being dropped (assuming that the 2.5% of dropped packets are within the measurement error). At least, this is true for UDP. We still need to investigate the differences for TCP equivalent transmissions.
{{table_drop_hbh}} lists all experiments (types and lengths) along with their success percentages, i.e., packets with a hop-by-hop header reaching their destination, both for UDP and TCP:
| Option Type | Length (bytes) | UDP | TCP | | Skip | 8 | 8.6% | 9.1% | | Discard | 8 | 0.0% | 0.0% | | Skip | 256 | 2.4% | 2.4% | | Skip w/ PadN| 256 | 0.5% | 0.6% | | Discard | 256 | 0.0% | 0.0% | | Skip | 512 | 1.4% | 1.5% | | Discard | 512 | 0.0% | 0.0% | {: #table_drop_hbh title="Hop-by-hop Header Transmission"}
It appears that hop-by-hop options headers cannot reliably traverse the global Internet; only small headers with 'skipable' options have some chances. If the unknown hop-by-hop option has the 'discard' bits, it is dropped per specification, although we observed in some cases that such packets were not necessarily dropped directly by the very first hop. Globally, there are no notable differences between UDP and TCP.
{{table_drop_do}} lists all lengths that have been tested along with their success percentages, i.e., packets with a destination header reaching their destination, both for UDP and TCP:
| Length (bytes) | UDP | TCP | | 8 | 97.8% | 94.3% | | 16 | 97.7% | 90.5% | | 24 | 97.6% | 89.8% | | 32 | 93.5% | 86.2% | | 40 | 93.9% | 86.2% | | 48 | 93.7% | 86.1% | | 56 | 93.8% | 52.7% | | 64 | 45.9% | 37.8% | | 128 | 10.9% | 10.9% | | 256 | 4.3% | 4.3% | | 256 w/ PadN | 4.3% | 4.3% | | 512 | 3.1% | 3.1% | {: #table_drop_do title="Destination Header Transmission"}
The measurement revealed no difference with the discard bits, which tends to show that routers do not look inside the destination header, as expected.
The size of the destination options header has a major impact on the drop probability. It appears that destination headers larger than 24 octets already cause drops. It may be because the 40 octets of the IPv6 header + the 24 octets of the extension header (total 64 octets) is still in the limits of some router hardware lookup mechanisms while the next measured size (extension header size of 32 octets for a total of 72 octets) is beyond the hardware limit and some ASs have a policy to drop packets where the TCP/UDP ports are unknown. A major drop also occurs once the size reaches 64 bytes for UDP while, surprisingly, it happens at 56 bytes for TCP. In either case, the chances of surviving are approximately halved. We still need to investigate the differences for TCP equivalent transmissions.
The propagation of two kinds of fragmentation headers was analysed: atomic fragment (offset == 0 and M-flag == 0) and plain first fragment (offset == 0 and M-flag == 1). The {{table_drop_frag}} displays the propagation differences.
| M-flag | UDP | TCP | | 0 (atomic) | 55.8% | 49.8% | | 1 | 89.2% | 87.6% | {: #table_drop_frag title="IPv6 Fragments Transmission"}
The size of the overall IPv6 packets (512, 1280, and 1500 octets) has no major impact on the propagation.
{{table_no_drop}} lists ASs that do not drop transit traffic with extension headers and therefore follow the recommendations of {{?RFC9288}}:
{::include ./no_drop.inc} {: #table_no_drop title="ASs not dropping packets with Extension Headers"}
Measurements also include two protocol numbers that are mainly new use of IPv6 as well as AH and ESP. {{table_special_next_header}} indicates the percentage of packets reaching the destination.
| Next Header | Transmission | | NoNextHeader (59) | 98.2% | | Ethernet (143) | 98.3% | | Authentication (AH)| 98.1% | | ESP | 98.3% | {: #table_special_next_header title="Transmission of Special IP Protocols"}
The above indicates that those IP protocols can be transmitted over the global Internet without being dropped (assuming that the 2% of dropped packets are within the measurement error). Globally, there are no notable differences between UDP and TCP, for cases where it applies.
While the analysis has areas of improvement (geographical distribution and impact on latency), it appears that:
-
AH, ESP, and non-atomic fragmentation headers (to some extent) can traverse the Internet;
-
only routing headers types 0, 1 and 4 experiment problems over the Internet, other types have no problems;
-
hop-by-hop options headers do not traverse the Internet, whatever the size;
-
destination options headers are not reliable enough when it exceeds 24 octets.
Of course, the next phase of measurement with non-collaborating parties will probably give another view.
While active probing of the Internet may be considered as an attack, this measurement was done among collaborating parties and using the probe attribution technique described in {{?I-D.draft-vyncke-opsec-probe-attribution}} to allow external parties to identify the source of the probes if required.
This document has no IANA actions.
--- back
{:numbered="false"}
The authors want to thank AfriNIC, Angani, China Telecom, Jared Mauch, Sander Steffann, XiPeng Xiao, and Jan Zorz for allowing the free use of their labs. Other thanks to Ben Campbell and Fernando Gont who indicated a nice IPv6 hosting provider in Africa and South America.
Special thanks as well to Professor Benoît Donnet for his support and advices. This document would not have existed without his support.