How much overhead do HTTP headers add on average?
tldr: Testing 1000 popular domains for HTTP header sizes. On average it will take 1-2KB. There are small header with 300 byte and huge one with 28KB.
Introduction
Several month ago I made a talk in the local developer community for flutter developers about cartoonish representation of how low level networking works. The talk discussed networking mostly from frontend perspective and covered essential basics like DNS, TCP/IP, HTTP, TLS and etc. During the talk we calculated “optimal” page size limit based on MTU size, TCP/IP header sizes and congestion. So (MTU minus TPC/IP headers) * 10 packets is about 14 KB. But between TCP/IP headers and actual documents that we want to send (HTML, JSON, PNG, etc) there is another layer that is not accounting here. It is HTTP or more more specifically HTTP headers. Since it was just a sidenote of the talk and not part of the main discussion I just quickly mentioned that there is HTTP header that should be accounted for and just skipped it.
It also tickled my own curiosity. I know that HTTP header is dynamic (and let’s be honest in the current age of multi-megabyte pages is not really relevant) we can’t just say how big it is. But I wanted to explore how are big or small they are on average on popular sites. So here is my small experiments and explorations with popular pages.
Setup
The setup is pretty simple. We take some amount of popular web pages. Download them, split to HTTP header and content part and do some simple statistics/graphs to get some understanding of the topic. Since there are billion pages we need some limitations. I chose to take about 1000 most popular pages. This should be enough to get basic statistics on what is used on most popular pages.
For this I took the most popular 500 pages from Moz top 500. These are popular based on domain authority or how many backlinks they got from other websites. Also I took another 1000 most popular pages from DomCom: Top 10 million Websites These are popular based on Open page rank. I combined these into one list and filtered out duplicates which resulted in 1226 different domains. I will used use these to download their index pages and calculate header/body sizes based on them. Here are data files if you want to explore yourself:
First attempt
I was already was playing around with dart and exploring some dart socket and how to use it over proxies. So I wrote a simple dart program that we connecting through a list of proxies to a web page and downloading it over HTTP 1.1. A of pages from the list were redirecting and I was saving both initial page and all intermediary redirects all the way up to the last resolving page.
It was working fine but I noticed that facebook was returning only “Unsupported browser” page. After trying different header types to mimic regular browsers I found out that facebook does not work over HTTP versions higher than 1.1. It is not a big deal but I thought if there will be a lot of pages like that they might not redirect to ‘unsupported browser’ but rather silently fail and to verify that I would need to go over outputs of all pages and that will take a bit than allocated time for this task. So instead of implementing custom HTTP/1.2+ or rewriting with darts ‘http’ package I decided to just go with ‘curl’. (I usually use python for scraping tasks but wanted to explore something different this time).
Second attempt
First I started a simple bash script that load a file, iterates over all urls, downloads each page and saves needed information.
curl -s -L -o /dev/null "$effective_url" \
--connect-timeout 10 \
--max-time 30 \
-w "%{size_header}, %{size_download}, %{http_code}, %{url_effective}, ${url}\n" \
-H "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
-H "Accept-Language: en-US,en;q=0.5" \
-H "Connection: keep-alive" \
-H "Upgrade-Insecure-Requests: 1" \
-H "Sec-Fetch-Dest: document" \
-H "Sec-Fetch-Mode: navigate" \
-H "Sec-Fetch-Site: none" \
-H "Sec-Fetch-User: ?1" \
-H "Priority: u=0, i" \
-H "Pragma: no-cache" \
-H "Cache-Control: no-cache" \
# -H "Accept-Encoding: gzip, deflate, br, zstd" \
The main trick here is -w "%{size_header}, %{size_download}, %{http_code}, %{url_effective}, ${url}\n"
Curl gives us the ability to output some meta information about document download.
Here I save header size, body size, status code, resulted url and initial url.
Initially I wanted save specifically final page after all redirects (thus -L
option). But after getting data for all urls and inspecting output I found that
for some pages HTTP header sizes were unusually big. Comparing those headers with
regular browser output I found that curl give total size for all headers for all
redirected pages. This is not what we wanted.
Third (final) attempt
After different experiments and failures I came up with a solution suitable for me.
First of all I used the same above curl request but for output used this:
-w "%{url_effective}"
And capture it into a variable effective_url. This will do all redirects and
give back the final resolved url. Then this url is used for second request but
use this effective_url. In second request I capture just **%{http_code}. Header
and body are saved in separate files in specific folder with these additional
curl parameters:
-D $HEADER -o $BODY
Where $HEADER and $BODY are just unique (per url) file paths where actual HTTP
headers and body is saved. And then just output data with:
header_size=$(wc -c < $HEADER)
body_size=$(wc -c < $BODY)
message="$header_size, $body_size, $http_code, $effective_url, $url"
Sample output for running this script looked like this:
...
1756, 87692, 200, https://www.bigcommerce.com/, https://bigcommerce.com
1649, 131923, 200, https://www.bild.de/, https://bild.de
558, 2027, 202, https://www.binance.com/, https://binance.com
3225, 41678, 200, https://www.bing.com:443/?toWww=1&redig=0A92B04D92B84006AEA2F88595AEFAB5, https://bing.com
220, 269, 200, https://bingol.edu.tr/, https://bingol.edu.tr
...
Some of the pages returned statuses other than 200. It is probably because of initially proxies and later because script was running on a webserver and it look like they flagged requests as inappropriate based on IP. It is fine because first of all they are not many of them and second it will give some data to explore how header/body sizes change on non 200 responses.
Results
There have been different attempts with incorrect or misleading results. Here are final raw data after checks and cleaning.
- Curl 500url compressed
- Curl 500url not compressed
- Curl 1000url compressed
- Curl 1000url not compressed
- Curl all urls compressed
- Curl all urls not compressed
- raw socket 500 not compressed
- raw socket 1000 not compressed
- raw socket all not compressed
First of all if we look at header sizes for all data gotten with curl we get this graph.
This graph in its raw format is a bit misleading. First of all there are outliers like zero sized header and a few 10KB+ headers. It also contains non 200 responses that contribute to tiny and zero header sizes. Here are bottom and top 10 results:
0 0 https://acidholic.com/ https://acidholic.com
0 0 https://akamaihd.net/ https://akamaihd.net
0 0 https://alicdn.com/ https://alicdn.com
0 0 https://amazonaws.com/ https://amazonaws.com
0 0 https://azurefd.net/ https://azurefd.net
0 0 https://bcg.com/ https://bcg.com
0 0 https://bls.gov/ https://bls.gov
0 0 https://bp.blogspot.com/ https://bp.blogspot.com
0 0 https://citeseerx.ist.psu.edu/ https://citeseerx.ist.psu.edu
0 0 https://citotoetsgroep4.nl/ https://citotoetsgroep4.nl
...
...
8175 200 https://www.engadget.com/ https://www.engadget.com
8501 200 https://outlook.live.com/mail/0/ https://outlook.live.com/mail/0
9491 200 https://www.tiktok.com/explore https://www.tiktok.com/explore
9504 307 https://rumble.com/ https://rumble.com
9592 200 https://www.who.int/ https://www.who.int
9931 200 https://www.fifa.com/ https://www.fifa.com
11461 200 https://www.fao.org/home/en https://www.fao.org/home/en
11994 200 https://x.com/ https://x.com
14680 200 https://www.logitech.com/en-us https://www.logitech.com/en-us
18301 200 https://www.americanexpress.com/ https://www.americanexpress.com
Also from the above plot it is hard to see how sizes are distributed. Even though we can probably estimate that most results fit into 2500 byte upper limit let’s see into histogram.
From this histogram we can see that first 1000 datapoints fit into somewhere between 0 and 3500+ bytes. From now on I will be filtering out all 0 sized results (both for headers and for bodies).
Now let’s look at different slices of this information to get better view and intuition.
Here is a similar graph but showing only responses with status code of 200. It
is similar to previous but for histogram it has count of items in the bin and
upper bound value for that bin.
188 200 https://cargo.site/
198 200 https://reg.ru/
204 200 https://www.enable-javascript.com/
206 200 https://gutenberg.org/
206 200 https://planeta.ru/
216 200 https://sedoparking.com/infopage/en/index.html
220 200 https://bingol.edu.tr/
236 200 https://meyerweb.com/
238 200 https://home.pl/
242 200 https://phpscriptsmall.com/
...
...
8089 200 https://vimeo.com/
8175 200 https://www.engadget.com/
8501 200 https://outlook.live.com/mail/0/
9491 200 https://www.tiktok.com/explore
9592 200 https://www.who.int/
9931 200 https://www.fifa.com/
11461 200 https://www.fao.org/home/en
11994 200 https://x.com/
14680 200 https://www.logitech.com/en-us
18301 200 https://www.americanexpress.com/
Initially I though that small headers (200bytes) is for sites are basically placeholders pages for something. But it looks like they are regular web pages with content.
Now if we look at all headers with status code not equal to 200 we get this
graph:
We still get outliers but we can already that these types of pages have some clusters. Quite a few of them are around 2KB, another block is closer to 100-200 byte range. Here are top and bottom results:
163 404 https://googletagmanager.com/
182 404 https://static.googleusercontent.com/
215 400 https://i0.wp.com/
215 400 https://i1.wp.com/
215 400 https://i2.wp.com/
226 404 https://fonts.gstatic.com/
226 404 https://googleadservices.com/
226 404 https://www.gstatic.com/
226 404 https://pagead2.googlesyndication.com/
...
...
2238 403 https://help.openai.com/
2244 403 https://platform.openai.com/
2248 403 https://www.canva.com/
2425 403 https://sourceforge.net/
2649 403 https://www.getyourguide.com/
3300 403 https://www.oracle.com/
3466 403 https://www.godaddy.com/
3469 403 https://www2.hm.com/en_us/index.html
5235 403 https://chatgpt.com/
9504 307 https://rumble.com/
Here we see some “fake” pages which don’t contain normal page for it’s root document and just return 404 and there are some pages that didn’t like our IP and just gave us 403.
Now let’s remove outliers. I decided to remove 5% from each side which leaves with 90% of original data. It will remove smallest and largest values and give give us a clearer picture of how the majority looks like.
If we look at results with status other than 200 we can clearly see clusters at
around 250 byte and 1800 byte marks. But there are few data points so this
should be looked at carefully.
Interpretation and side notes
Initially we had 1226 domains/subdomains. We tried downloading their index document (index.html) and split them into separate HTTP headers and bodies. Since we were downloading just the final page after redirect we have left only 1128 urls. This is due some domains redirected to the same new domain.
These could be split into 936 links that returned status 200 and 149 links that returned status other than 200. If we filter out bottom and top 5% to remove extremums then we get 844 links (status 200) and 135 links (status != 200).
Let’s look at regular pages (bottom 5% and top 5% removed):
Mean: 1566
Median: 1149
Mode: 803
Std Dev: 1134
Variance: 1285151
Min: 390
Max: 5292
So in average HTTP header is 1.5KB and 50% percent of headers fit into 1KB. Standard deviation is big compared to mean and data is very spread out.
If we look at “error” pages with status codes other than 200 we get similar results but they are a bit smaller.
Mean: 1172
Median: 1147
Mode: 252
Standart Deviation: 723
Variance: 522230
Min: 226
Max: 2248
Since these are “error” pages it is normal that big portion of them have smaller headers.
These were averages but what about pages on the edges? Well bottom part is easy they are either zero or very small because server sent close to minimum required. On the top part we have pages that are 10KB+. In this dataset the biggest header in this test was https://www.americanexpress.com/ and it is 18301 bytes. Here are top 10:
8175 https://www.engadget.com
8501 https://outlook.live.com/mail/0
9491 https://www.tiktok.com/explore
9504 https://rumble.com
9592 https://www.who.int
9931 https://www.fifa.com
11461 https://www.fao.org/home/en
11994 https://x.com
14680 https://www.logitech.com/en-us
18301 https://www.americanexpress.com
Why do these pages have big HTTP headers?
Well quick look at all the pages with larger than average headers gives two main headers that give most impact on header size: content-security-policy and link
content-security-policy - CSP is an HTTP response header that tells the browser what kinds of resources (scripts, images, styles, etc.) are allowed to load on a web page. It’s one of the main defenses against Cross-Site Scripting (XSS) and data injection attacks.
So bigger companies setup more fine grained configuration on how browser should
allow different types of content to be downloaded. So a regular website might have
this strict setting for CSP: Content-Security-Policy: default-src 'self'; script-src 'self'; object-src 'none'; base-uri 'self'; frame-ancestors 'none'
But for example a x.com has CSP settings set to (I slightly changed how it look
by adding newlines for easy reading but it is in single line):
content-security-policy:
connect-src 'self'
blob: https://fonts.googleapis.com/css
https://mapsresources-pa.googleapis.com
https://maps.googleapis.com
https://translate.googleapis.com
https://www.gstatic.com/maps/
https://*.pscp.tv
https://*.twimg.com
https://*.video.pscp.tv
https://aa.twitter.com
https://aa.x.com
https://accounts.google.com/gsi/
https://ads-api.twitter.com
https://ads-api.x.com
https://api-stream.twitter.com
https://api-stream.x.com
https://api.twitter.com
https://api.x.ai
https://api.x.com
https://api.x.com
https://caps.twitter.com
https://caps.x.com
https://grok.x.com
https://jf.twitter.com
https://jf.x.com
https://jf-t.x.com
https://pay.twitter.com
https://pay.x.com
https://sentry.io
https://ton-staging.atla.twitter.com
https://ton-staging.atla.x.com
https://ton-staging.pdxa.twitter.com
https://ton-staging.pdxa.x.com
https://ton.twitter.com
https://ton.local.twitter.com
https://ton.x.com
https://twitter.com
https://upload.twitter.com
https://upload.x.com
https://www.google-analytics.com
https://x.com
https://grok-api.gcp.mouseion.dev
https://assets.mouseion.dev
https://grok.com
https://assets.grok.com
https://*.adtrafficquality.google
https://*.googlesyndication.com
https://*.doubleclick.net
https://adservice.google.com
https://www.googleadservices.com
https://pagead2.googlesyndication.com
https://www.google.com
https://google.com
https://via.intercom.io
https://api.intercom.io
https://api.au.intercom.io
https://api.eu.intercom.io
https://api-iam.intercom.io
https://api-iam.eu.intercom.io
https://api-iam.au.intercom.io
https://api-ping.intercom.io
https://nexus-websocket-a.intercom.io
wss://nexus-websocket-a.intercom.io
https://nexus-websocket-b.intercom.io
wss://nexus-websocket-b.intercom.io
https://nexus-europe-websocket.intercom.io
wss://nexus-europe-websocket.intercom.io
https://nexus-australia-websocket.intercom.io
wss://nexus-australia-websocket.intercom.io
https://uploads.intercomcdn.com
https://uploads.intercomcdn.eu
https://uploads.au.intercomcdn.com
https://uploads.eu.intercomcdn.com
https://uploads.intercomusercontent.com
https://production.plaid.com/
https://sandbox.plaid.com/
https://ingestion.dv.socure.io
https://network.dv.socure.io/
https://analytics.dv.socure.io/
https://payments-dev.x.com/customer/wasm/forward-with-v1.wasm
https://payments-staging.x.com/customer/wasm/forward-with-v1.wasm
https://payments-prod.x.com/customer/wasm/forward-with-v1.wasm
https://money-dev.x.com/customer/wasm/forward-with-v1.wasm
https://money-staging.x.com/customer/wasm/forward-with-v1.wasm
https://money.x.com/customer/wasm/forward-with-v1.wasm
https://api.stripe.com
https://m.castle.io
https://checkoutshopper-live.adyen.com
wss://*.pscp.tv
https://vmap.snappytv.com
https://vmapstage.snappytv.com
https://vmaprel.snappytv.com
https://vmap.grabyo.com
https://dhdsnappytv-vh.akamaihd.net
https://pdhdsnappytv-vh.akamaihd.net
https://mdhdsnappytv-vh.akamaihd.net
https://mdhdsnappytv-vh.akamaihd.net
https://mpdhdsnappytv-vh.akamaihd.net
https://mmdhdsnappytv-vh.akamaihd.net
https://mdhdsnappytv-vh.akamaihd.net
https://mpdhdsnappytv-vh.akamaihd.net
https://mmdhdsnappytv-vh.akamaihd.net
https://dwo3ckksxlb0v.cloudfront.net
https://media.riffsy.com
https://*.giphy.com
https://media.tenor.com
https://c.tenor.com
wss://chat-ws.x.com
https://d1muhwhmpsz4u8.cloudfront.net/
https://d2bchqfeno8n2m.cloudfront.net/
https://d2shtph9y6bxk.cloudfront.net/
https://xchat-hsm-staging.x.com/
https://realm-a.x.com
https://realm-b.x.com
https://realm-west1.x.com
https://realm-east1.x.com
https://hsm-staging.x.com
https://ads-twitter.com
https://analytics.twitter.com https://analytics.x.com ;
default-src 'self';
form-action 'self'
https://twitter.com
https://*.twitter.com
https://x.com
https://*.x.com
https://localhost.twitter.com:3443
https://localhost.x.com:3443
https://intercom.help
https://api-iam.intercom.io
https://api-iam.eu.intercom.io
https://api-iam.au.intercom.io;
font-src 'self'
https://*.twimg.com
https://js.intercomcdn.com
https://fonts.intercomcdn.com;
frame-src 'self'
https://accounts.google.com/
https://accounts.google.com/gsi/
https://cards-frame.twitter.com
https://cdn.plaid.com/
https://client-api.arkoselabs.com/
https://content.googleapis.com/
https://iframe.arkoselabs.com/
https://mobile.twitter.com
https://mobile.x.com
https://pay.twitter.com
https://pay.x.com
https://google.com
https://www.google.com
https://intercom-sheets.com
https://www.intercom-reporting.com
https://www.youtube.com
https://player.vimeo.com
https://fast.wistia.net
https://console.googletagservices.com
https://*.doubleclick.net
https://*.adtrafficquality.google
https://*.safeframe.googlesyndication.com
https://www.googleadservices.com
https://googleadservices.com
https://adservice.google.com
https://*.googlesyndication.com
https://td.doubleclick.net
https://payments-dev.x.com/
https://payments-staging.x.com/
https://payments-prod.x.com/
https://sdn.payments-dev.x.com/
https://sdn.payments-staging.x.com/
https://sdn.payments-prod.x.com/
https://money-dev.x.com/
https://money-staging.x.com/
https://money.x.com/
https://sdn.money-dev.x.com/
https://sdn.money-staging.x.com/
https://sdn.money.x.com/
https://p2pcreditcardiframesandbox.crbcos.com
https://p2pcreditcardiframe.crbcos.com
https://verify-sandbox.plaid.com/
https://*.js.stripe.com
https://js.stripe.com
https://hooks.stripe.com
https://cdn.getpinwheel.com/
https://artifacts.grokusercontent.com
https://twitter.com
https://x.com
https://recaptcha.net/recaptcha/;
img-src 'self'
blob: data:
https://www.google.com/maps/place/
https://imgs.search.brave.com
https://*.cdn.twitter.com
https://*.cdn.x.com
https://ton.twitter.com
https://ton.x.com
https://*.twimg.com
https://analytics.twitter.com
https://analytics.x.com
https://cm.g.doubleclick.net
https://www.google-analytics.com
https://maps.googleapis.com
https://www.periscope.tv
https://www.pscp.tv
https://ads-twitter.com
https://ads-api.twitter.com
https://ads-api.x.com
https://api.x.com
https://developer.x.com
blob: data:
https://js.intercomcdn.com
https://static.intercomassets.com
https://downloads.intercomcdn.com
https://downloads.intercomcdn.eu
https://downloads.au.intercomcdn.com
https://uploads.intercomusercontent.com
https://gifs.intercomcdn.com
https://video-messages.intercomcdn.com
https://messenger-apps.intercom.io
https://messenger-apps.eu.intercom.io
https://messenger-apps.au.intercom.io
https://*.intercom-attachments-1.com
https://*.intercom-attachments.eu
https://*.au.intercom-attachments.com
https://*.intercom-attachments-2.com
https://*.intercom-attachments-3.com
https://*.intercom-attachments-4.com
https://*.intercom-attachments-5.com
https://*.intercom-attachments-6.com
https://*.intercom-attachments-7.com
https://*.intercom-attachments-8.com
https://*.intercom-attachments-9.com
https://static.intercomassets.eu
https://static.au.intercomassets.com
https://media.riffsy.com
https://*.giphy.com
https://media.tenor.com
https://c.tenor.com
https://*.pscp.tv
https://*.periscope.tv
https://prod-periscope-profile.s3-us-west-2.amazonaws.com
https://platform-lookaside.fbsbx.com
https://scontent.xx.fbcdn.net
https://scontent-sea1-1.xx.fbcdn.net
https://*.googleusercontent.com
https://t.co/1/i/adsct
https://*.googleusercontent.com
https://*.gstatic.com
https://*.googlesyndication.com
https://*.adtrafficquality.google
https://www.google.com/ads/measurement/
https://*.google.com/ads/measurement/
https://googleads.g.doubleclick.net
https://google.com
https://www.google.com
https://plaid-merchant-logos.plaid.com
https://plaid-counterparty-logos.plaid.com
https://assets.mouseion.dev
https://assets.grok.com;
manifest-src 'self';
media-src 'self' data: blob:
https://twitter.com
https://x.com
https://*.twimg.com
https://*.vine.co
https://*.pscp.tv
https://*.video.pscp.tv
https://js.intercomcdn.com
https://downloads.intercomcdn.com
https://downloads.intercomcdn.eu
https://downloads.au.intercomcdn.com
https://dhdsnappytv-vh.akamaihd.net
https://pdhdsnappytv-vh.akamaihd.net
https://mdhdsnappytv-vh.akamaihd.net
https://mdhdsnappytv-vh.akamaihd.net
https://mpdhdsnappytv-vh.akamaihd.net
https://mmdhdsnappytv-vh.akamaihd.net
https://mdhdsnappytv-vh.akamaihd.net
https://mpdhdsnappytv-vh.akamaihd.net
https://mmdhdsnappytv-vh.akamaihd.net
https://dwo3ckksxlb0v.cloudfront.net;
object-src 'none';
script-src 'self' 'unsafe-inline'
https://maps.googleapis.com
https://*.twimg.com
https://recaptcha.net/recaptcha/
http://www.gstatic.com/cast/sdk/libs/caf_receiver/v3/cast_receiver_framework.js
https://accounts.google.com/gsi/client
https://apis.google.com/js/api.js
https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js
https://client-api.arkoselabs.com/
https://static.ads-twitter.com
https://twitter.com
https://www.google-analytics.com
https://www.gstatic.com/cast/sdk/libs/caf_receiver/v3/cast_receiver_framework.js
https://x.com
https://sdn.payments-dev.x.com/assets/loader.min.js
https://sdn.payments-staging.x.com/assets/loader.min.js
https://sdn.payments-prod.x.com/assets/loader.min.js
https://sdn.money-dev.x.com/assets/loader.min.js
https://sdn.money-staging.x.com/assets/loader.min.js
https://sdn.money.x.com/assets/loader.min.js
https://sdk.dv.socure.io/latest/device-risk-sdk.js
https://cdn.plaid.com/link/v2/stable/link-initialize.js
https://payments-dev.x.com/customer/wasm/xxp-forward-with-sdk.js
https://payments-staging.x.com/customer/wasm/xxp-forward-with-sdk.js
https://payments-prod.x.com/customer/wasm/xxp-forward-with-sdk.js
https://money-dev.x.com/customer/wasm/xxp-forward-with-sdk.js
https://money-staging.x.com/customer/wasm/xxp-forward-with-sdk.js
https://money.x.com/customer/wasm/xxp-forward-with-sdk.js
https://js.stripe.com
https://*.js.stripe.com
https://cdn.getpinwheel.com/pinwheel-v3.1.0.js
https://securepubads.g.doubleclick.net
https://www.googletagservices.com
https://*.googletagservices.com
https://pagead2.googlesyndication.com
https://adservice.google.com
https://www.googleadservices.com
https://ads.google.com
https://tpc.googlesyndication.com
https://*.tpc.googlesyndication.com
https://www.google.com
https://googleads.g.doubleclick.net
https://app.intercom.io
https://widget.intercom.io
https://js.intercomcdn.com
'wasm-unsafe-eval' 'nonce-MjIxYzQzMjktMGJmMi00YWNhLTg4MmUtMDNiMjhiZTRmMTg1';
style-src 'self' 'unsafe-inline'
https://accounts.google.com/gsi/style
https://*.twimg.com;
worker-src 'self' blob:; report-uri https://x.com/i/csp_report?a=O5RXE%3D%3D%3D&ro=false
Second “offender” is link header in the HTTP response.
Link header in HTTP is used by servers to provide relationships between resources. It can convey information like: Pagination links, Related resources, Preloading or prefetching instructions, Canonical URLs.
This is used to help browsers to preload some resources, might help with SEO optimization etc. Some websites don’t use it at all but other use it quite extensively. Here is how it looks for www.logitech.com (which has second largest header in our set):
link:
<./_app/immutable/assets/0.BmUwg0OD.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Tooltip.CIt8wLqd.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/TextField.BO3MU91H.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Picture.uZYgJ1kR.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Modal.BA7QV53T.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Drawer.BrNLa59s.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/OrderStatus.C_W6J39V.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/SwatchGroup.BIXu1SoU.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Badges.CuhXVqRm.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Skeleton.ByVz-TnP.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/ProductCard.ChIpM_2K.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/DropdownSelect.Dx4C0AFL.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/EmblaCarousel.DeZlrFLh.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/GridColumns.BLhBfQ1D.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/MyAccountAccordion.Di-g4IEv.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/NotifyMe.4I_RbR00.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/RibbonMessage.V6U_ZgGI.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/ProductSetModal.BesSfSWX.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/2.BjqyT-KM.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Gift.Dg0qapQA.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/LegalPage.D38gJJjk.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/GiftGuidePlp.DELFs5n8.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/WheelConfigurator.DvLw9yMs.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/PlayButton.9m-L7Ujp.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/EditorialCards.D7aqCbSh.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Configurators.DSjYh2FM.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/CategoryCards.bx2JviTG.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/BackgroundBanner.BEGwdjAs.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/SeeMore.DHvzwpve.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Faqs.Su8zDiUM.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/ReasonsToBuy.Dvaw2BUv.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Breadcrumb.DiRfzvE-.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/QuickLinks.BhJZGJ6R.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Card.DY6KeWXA.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/BladeIntro.BjWOuH9Z.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/StandardCards.DJ4U0L2o.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/ProductGrid.CBYoD17i.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/FilterBar.B2A9O1Tg.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Bottomline.CWWui-fC.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/ProductBanner.wLQs3Mvm.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/BoutiqueHeroBanner.omPf8uR1.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Checkerboard.CnUcPnhG.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/JumbotronAndMediaCards.BhkDG4XB.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/Highlights.cLdA_vzO.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/AudioPlayer.Dnx6PVqL.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/MediaCards.4hcQMSeV.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/ImageGallery.ClV1Q7uT.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/FeaturedSlideshow.CIsn24t1.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/EmailCapture.s7vQIoQl.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/BackgroundImage.DwlZo5tE.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/InteractiveHeroBanner.DxragpLp.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/PromoSlideshow.Cgz8DbdB.css>; rel="preload";as="style"; nopush,
<./_app/immutable/assets/CountdownBanner.CpnrTeAh.css>; rel="preload";as="style"; nopush,
<./_app/immutable/entry/start.Bo9X7LCa.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DkX4V_4r.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/vqdJvOLt.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/ae09DoGu.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/D0C9qiCV.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CTFNgzZY.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DpaXFcUM.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/owOoXXu1.js>; rel="modulepreload"; nopush,
<./_app/immutable/entry/app.3XqFV2zw.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Dp1pzeXC.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CKLPxLCL.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CWj6FrbW.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/pHaOfS12.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CgSAfupW.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Xogsordm.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BifgfUVB.js>; rel="modulepreload"; nopush,
<./_app/immutable/nodes/0.Dzkrss74.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/6Nlpsswe.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Bgwlalhc.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Bl7uWzAQ.js>; rel="modulepreload"; nopush,
<./_app/immutable/nodes/2.Ds8jGgVp.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/D3MoPhG0.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Jcrk9AUK.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CwSf_6Oo.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DFfcXFJY.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DR26F1ww.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/svCTgsbl.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CxeSUyL0.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DgxvQK41.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CnsIcpcx.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CPKwXkwV.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/COaciFmb.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DnRILV_f.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CYIqOY2g.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/toVdKOst.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DC0bqrkQ.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BtKAxfA3.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BWHp0Vsr.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/De6nu5MR.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BDcPnI-Z.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DiIMQyML.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BF6Ytkcr.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Bt-4VhZM.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Csbk5jSW.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Cg-ONzKS.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Dme6TEhe.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DwAdCQdj.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BfLEW3ss.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BLeZixJ6.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B9jBDwK_.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/IJVLHI5p.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DcihOCFz.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/gqeLmaaO.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DR0-zYLf.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B8gu9aFp.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/H_q5vUyg.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/D6pw74BZ.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CD4MRKs4.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CcRZZGcE.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CPGb2Ha8.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CH7BE6MN.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CLw_WQ5l.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BFADZywx.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CJ01Vb4c.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DysaiVgm.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/zGzYgxqd.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CWmzcjye.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/C7CoQe0o.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CuEM3MOL.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BVvXFXiJ.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/PT7ERBxL.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/htl6zqkE.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CtaOvT8R.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BBmOzkJw.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BOyEWY4K.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/C1gpZCO_.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/E1e8QwvJ.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Cycr3KLS.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BoKTXoh4.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BKpiWXUB.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BSLEhssX.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BAAlleYb.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B8B85e8Q.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/_1l_qwxM.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DSlhB7PO.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/SznIFsVG.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/yROr9UHf.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BIcwaRCq.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BaZXYk2g.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DIEOYCeo.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CDk7q2ng.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Gv5466mM.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CWmWDg4N.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/LVnR6bjH.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BZ6IGNqI.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/JdOuQZ8H.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/wG8eApaj.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/aBPCaKf1.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DXsNBmiM.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Bri3zkjy.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CIC-jhmz.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BNr6liSM.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/SiraxVBd.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/j3SyHAGr.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/a9Tn--oZ.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/QyMysE8m.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DkK28eBK.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CZcxppUj.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B4P5_jGO.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Bny1nKgg.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BwiTatk4.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Dqgy_uMd.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B_fHgl1t.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/OBgJsU14.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BK_KtU3Q.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BWTJsKEf.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/_TnfFlQp.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BGRsod_p.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Bz58EIw4.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CFPu4TTR.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B6b3zP9u.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B0tIWchx.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/FAk7kFvN.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B8OxPEOE.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/F1Ofg1lG.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/SgK5LUoG.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/B2Lz-WBb.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CejpDeAX.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CTndWtUC.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/kDFV6buO.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BcXaC3iu.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DcOYx2D6.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BkUPqegt.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/BEJZVMde.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/DdBsBbbE.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CNQS9sZG.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/Di1fwpOV.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/dryfmCJt.js>; rel="modulepreload"; nopush,
<./_app/immutable/nodes/6.DyXqgdxg.js>; rel="modulepreload"; nopush,
<./_app/immutable/chunks/CuEwkyQi.js>; rel="modulepreload"; nopush
Some sites might use both of this headers and you can see how it can easily get bigger. For example ‘finance.yahoo.com’ uses both.
This is 28KB HTTP header. I got it using a basic socket through proxy. Getting this page through other methods gives other sizes. For example downloading it through the browser gave 15KB header size.
Compression
The sizes that we discussed in this article are all without compression. They are raw, not compressed data sizes. This is what we can expect from HTTP/1.1 because it does not support header compression. It has only compression for body content.
HTTP2 and HTTP3 were designed around to reduce HTTP overhead and so their binary format support compression headers. HTTP2 uses HPACK and HTTP3 uses QPACK. In this post I didn’t explore those but I think it is safe to assume that for larger HTTP headers could provide 40-60% size reduction. Unfortunately I didn’t have easy to use tools at hand to explore that area. Maybe we could look into it in one of the future posts.
Conclusion
HTTP header sizes usually not included in the approximation of budget for page sizes. It is hard to calculate exactly due to its very dynamic nature and each website and page might have wide variety of sizes from very small ones with 250 byte to huge 28KB headers. But averaging most popular websites we can have a rough estimate of 1-2KB for regular HTTP header.