ERNIE-AI-Developer-Challenge
An Early Benchmark of Quality of Experience Between HTTP/2 and HTTP/3 using Lighthouse
Darius Saif $ ^{*} $ , Chung-Horng Lung $ ^{\dagger} $ , Ashraf Matrawy $ ^{\ddagger} $
Carleton University, Department of Systems and Computer Engineering
Email: Dariussaif $ ^{*} $ , Chlung $ ^{\dagger} $ , Amatrawy $ ^{\ddagger} $ @sce.carleton.ca
Abstract—Google’s QUIC (GQUIC) is an emerging transport protocol designed to reduce HTTP latency. Deployed across its platforms and positioned as an alternative to TCP+TLS, GQUIC is feature rich: offering reliable data transmission and secure communication. It addresses TCP+TLS’s (i) Head of Line Blocking (HoLB), (ii) excessive round-trip times on connection establishment, and (iii) entrenchment. Efforts by the IETF are in progress to standardize the next generation of HTTP’s (HTTP/3, or H3) delivery, with their own variant of QUIC. While performance benchmarks have been conducted between GQUIC and HTTP/2-over-TCP (H2), few analyses, to our knowledge, have taken place between H2 and H3. In addition, past studies rely on Page Load Time as their main, if not only, metric. The purpose of this article is to benchmark the latest draft specification of H3 and dig into a user’s Quality of Experience (QoE) by using Lighthouse: an open source (and metric diverse) auditing tool. Our findings show that, for one of H3’s early implementations, H3 is mostly worse but achieves a higher average throughput. Index Terms—Benchmarking. QUIC. HTTP/3. Lighthouse.
I. INTRODUCTION
QUIC is an emerging transport protocol which has been developed, and rolled out across services, by Google $ [1] $ . Its features akin to TCP+TLS (such as loss and congestion control, security $ [2] $ , Forward Error Correction (FEC) $ [3] $ , and even multi-path $ [4] $ ) position QUIC as an alternative to the former two. QUIC also brings advanced features like stream multiplexing to the table.
The primary motivation for QUIC is to reduce web page latency, thus bolstering a user’s Quality of Experience (QoE) $ ^{[5]} $ . QUIC’s major advantages over TCP+TLS are (i) eliminating Head-of-Line Blocking (HoLB) through stream multiplexing of independent resources, and (ii) fewer Round-Trip Times (RTTs) required on connection establishment, thanks to QUIC’s cross-layer design. Google researchers have proposed a disruptive approach rather than extensions to TCP most notably because of TCP’s entrenchment in networks and Operating Systems (OS). Rather, QUIC is rapidly deployable, as it runs in user space.
The IETF has begun standardizing their own variant of QUIC. This transport has become the backbone of the next generation protocol HTTP/3 (H3) [6], whereby HTTP semantics are mapped over QUIC. A comparison of HTTP/3 and HTTP/2 stacks is illustrated in Figure 1. Given this, Google’s implementation is now commonly referred to as GQUIC.
Performance comparisons between (G)QUIC and TCP+TLS have primarily considered Page Load Time (PLT). This article’s purpose is to extend upon those analyses from the
standpoint of providing better visibility on QoE. As such, use of Lighthouse $ ^{[7]} $ is proposed. It is an open source auditing tool which provides information-rich metrics and an aggregate performance score. Lighthouse is able to capture QoE features (like HoLB and prioritization) which a PLT analysis cannot.
This article provides four main points of contribution: (i) an early look at H3’s performance, which has taken place in one other study $ [8] $ , to our knowledge, (ii) comparison against HTTP/2 (H2) over TCP+TLSv1.3, which is more competitive than TCP+TLSv1.2 in terms of connection establishment, (iii) a discussion on how the differences between GQUIC and IETF QUIC may affect their respective performance, and (iv) incorporating more metric diversity into test scenarios to better represent QoE implications, not widely considered before.
Studies on GQUIC [9], [10] found it to be more suitable than H2-over-TCP+TLSv1.2 in networks with high RTT. Our study between H3 (with IETF QUIC, hereby simply called QUIC) and H2-over-TCP+TLSv1.3 did not yield the same observation. Explanations to this are offered in this article and we invite others to reproduce, and confirm, the results. Our benchmarking was performed on Chrome Canary to an NGINX server with a custom CloudFlare patch to support H3.
The rest of this article is organized as follows: Section II surveys related works in this area. Details of the setup and metrics used are covered in Sections III and IV, respectively. Then, the benchmarking methodology and results are presented in Section V and VI. Discussion on the results and the article’s conclusions are made in Sections VII and VIII. Finally, future works are presented in Section IX.