Maintainers:Fastly: Difference between revisions

(11 intermediate revisions by 3 users not shown)

Line 1:

[https://www.fastly.com Fastly] is a global CDN provider that powers [https://cache.nixos.org https://cache.nixos.org], one of our mission-critical services, through ~~the~~ [https://www.fastly.com/open-source Open Source and Non-Profit Program].

[https://www.fastly.com Fastly] is a global CDN provider that powers [https://cache.nixos.org https://cache.nixos.org], one of our mission-critical services, through their [https://www.fastly.com/open-source Open Source and Non-Profit Program].

This page gives some basic details about what the configuration for our services looks like. In the future, we hope to integrate more [https://nixos.org https://nixos.org] services with Fastly, such as [https://hydra.nixos.org Hydra], and the main homepage.

Line 7:

== Configuration details ==

The core configuration details for our services are located in [https://github.com/nixos/~~fastly-config fastly-config~~] ~~('''TODO: LINK FIXME''')~~, which you can quickly clone with git:

The core configuration details for our services are located in [https://github.com/nixos/infra infra], which you can quickly clone with git:

{{Commands|$ git clone https://github.com/nixos/~~fastly-configs~~}}

Check the <code>README.md</code> for details about the structure of the project, how to make and contribute changes, etc. It also describes the rough architecture of the integration(s).

Line 20:

** Better backbone routing: inter-POP network thanks to [https://docs.fastly.com/guides/performance-tuning/shielding shielding].

** Improved support for large NARs, by streaming results directly from S3 rather than "buffering" them in the POP first. This improves TTFB dramatically.

* Aggressive 404 caching, helping reduce the cost of low ~~negative~~ TTLs ~~on S3~~. ('''STATUS:''' '''MOSTLY DONE''')

* Aggressive 404 caching, helping reduce the cost of misses on S3, which will be very common, especially if users have low TTLs or Hydra is lagging. ('''STATUS:''' '''MOSTLY DONE''')

** Mapping 403s to proper 404s and caching them ('''STATUS:''' '''DONE''')

** Cache aggressive: ~1 month. ('''STATUS:''' '''NOT DONE''' -- requires upstream Hydra tooling changes, so cache uploads have their potential 404s purged in a timely manner.)

Line 49:

You should be set. This uses the real upstream nixos.org binary cache as a backend, so it should basically be up to date with <code>cache.nixos.org</code>.

==== Changelog ====

We'll also have a changelog recording any major upgrades made to the service. You can view the current one here: https://aseipp.notion.site/07c3be3df9f24d829471c6f8208a8570?v=41a80a849bb646d6a184bd1ce770edc4

==== Beta + IPv6 + HTTP/2 ====

Line 59:

Line 63:

Note the difference in the DNS name: <code>global.ssl</code> vs <code>freetls</code>.

==== Beta Issues ====

There are some known deficiencies with the beta, listed below:

* '''Any user can purge cache objects with no authentication'''. Use <code>curl -v -X PURGE https://<SOME URL></code> in order to do so. This is useful for debugging user issues, but during final deployment, we'll want to turn this off.

* '''Overly-conservative URL blocking'''. <s>The current implementation will only allow you to download <code>.narinfo</code>, <code>.ls</code>, and <code>.nar.xz</code> files -- this is to eliminate spurious/invalid requests to S3 for objects which could never possibly exist. If you see a 403 error returned from the server, then this is why. This should mean "recent" (few year old) evaluations should work fine -- ever since we've been using LZMA. This will be rectified in the future, but should only be noticeable to users on old channels.</s> '''FIXED''': This is now taken care of, and several other paths were fixed as well.

** We'll be sure to check the S3 metadata so that all filetypes in the cache can be downloaded properly, before final deployment.

* '''Origin connections do not use TLS'''. When connecting to a Fastly POP, you use TLS. When Fastly POPs talk to each other, they also use TLS. When a POP talks to S3, '''the beta service does not use TLS''' -- it talks to S3 over HTTP. This is due to a limitation in a feature we use called '''[https://docs.fastly.com/guides/performance-tuning/streaming-miss Streaming Miss]''', which is vital in reducing <abbr title="Time To First Byte">TTFB</abbr> for large, uncached objects. (Without it, a POP must download an entire, possibly multi-hundred-MB NAR file before it can begin serving you. Streaming miss allows your download to start instantly.) Support for streaming miss with TLS origins is currently deployed in "Limited Availability" for Fastly customers. <s>We'll be applying to the LA program for TLS Origin support before deploying to production, and testing it carefully.</s> '''FIXED''': This is now taken care of -- the final, live deployment will use TLS Origins! The beta currently does not.

== Known issues ==

Line 70:

Line 83:

=== IPv6 shenanigans ===

See [https://github.com/NixOS/nixpkgs/issues/65015]. Some users report that turning off IPv6 helps download things from <code>cache.nixos.org</code>. See also on the Fastly support forums: [https://support.fastly.com/hc/en-us/community/posts/360040169531-I-often-can-t-access-Fastly-servers-using-HTTPS-IPv6-RST-packets-received I (often) can't access Fastly servers using HTTPS+IPv6: RST packets received]. It is unclear how widespread this issue might be. Using an IPv4 only DNS CNAME may mitigate this in the long run.

See [https://github.com/NixOS/nixpkgs/issues/65015]. Some users report that turning off IPv6 helps download things from <code>cache.nixos.org</code>. See also on the Fastly support forums: [https://web.archive.org/web/20221208122000/https://support.fastly.com/hc/en-us/community/posts/360040169531-I-often-can-t-access-Fastly-servers-using-HTTPS-IPv6-RST-packets-received I (often) can't access Fastly servers using HTTPS+IPv6: RST packets received]. It is unclear how widespread this issue might be. Using an IPv4 only DNS CNAME may mitigate this in the long run.

== Future plans ==

@@ Line 1: / Line 1: @@
-[https://www.fastly.com Fastly] is a global CDN provider that powers [https://cache.nixos.org https://cache.nixos.org], one of our mission-critical services, through the [https://www.fastly.com/open-source Open Source and Non-Profit Program].
+[https://www.fastly.com Fastly] is a global CDN provider that powers [https://cache.nixos.org https://cache.nixos.org], one of our mission-critical services, through their [https://www.fastly.com/open-source Open Source and Non-Profit Program].
 This page gives some basic details about what the configuration for our services looks like. In the future, we hope to integrate more [https://nixos.org https://nixos.org] services with Fastly, such as [https://hydra.nixos.org Hydra], and the main homepage.
@@ Line 7: / Line 7: @@
 == Configuration details ==
-The core configuration details for our services are located in [https://github.com/nixos/fastly-config fastly-config] ('''TODO: LINK FIXME'''), which you can quickly clone with git:
+The core configuration details for our services are located in [https://github.com/nixos/infra infra], which you can quickly clone with git:
-{{Commands|$ git clone https://github.com/nixos/fastly-configs}}
+{{Commands|$ git clone https://github.com/nixos/infra}}
 Check the <code>README.md</code> for details about the structure of the project, how to make and contribute changes, etc. It also describes the rough architecture of the integration(s).
@@ Line 20: / Line 20: @@
 ** Better backbone routing: inter-POP network thanks to [https://docs.fastly.com/guides/performance-tuning/shielding shielding].
 ** Improved support for large NARs, by streaming results directly from S3 rather than "buffering" them in the POP first. This improves TTFB dramatically.
-* Aggressive 404 caching, helping reduce the cost of low negative TTLs on S3. ('''STATUS:''' <span style="color:#800080">'''MOSTLY DONE'''</span>)
+* Aggressive 404 caching, helping reduce the cost of misses on S3, which will be very common, especially if users have low TTLs or Hydra is lagging. ('''STATUS:''' <span style="color:#800080">'''MOSTLY DONE'''</span>)
 ** Mapping 403s to proper 404s and caching them ('''STATUS:''' <span style="color:#009000">'''DONE'''</span>)
 ** Cache aggressive: ~1 month.  ('''STATUS:''' <span style="color:#ff0000">'''NOT DONE'''</span> -- requires upstream Hydra tooling changes, so cache uploads have their potential 404s purged in a timely manner.)
@@ Line 49: / Line 49: @@
 You should be set. This uses the real upstream nixos.org binary cache as a backend, so it should basically be up to date with <code>cache.nixos.org</code>.
+==== Changelog ====
+We'll also have a changelog recording any major upgrades made to the service. You can view the current one here: https://aseipp.notion.site/07c3be3df9f24d829471c6f8208a8570?v=41a80a849bb646d6a184bd1ce770edc4
 ==== Beta + IPv6 + HTTP/2 ====
@@ Line 59: / Line 63: @@
 Note the difference in the DNS name: <code>global.ssl</code> vs <code>freetls</code>.
+==== Beta Issues ====
+There are some known deficiencies with the beta, listed below:
+* '''Any user can purge cache objects with no authentication'''. Use <code>curl -v -X PURGE https://<SOME URL></code> in order to do so. This is useful for debugging user issues, but during final deployment, we'll want to turn this off.
+* '''Overly-conservative URL blocking'''. <s>The current implementation will only allow you to download <code>.narinfo</code>, <code>.ls</code>, and <code>.nar.xz</code> files -- this is to eliminate spurious/invalid requests to S3 for objects which could never possibly exist. If you see a 403 error returned from the server, then this is why. This should mean "recent" (few year old) evaluations should work fine -- ever since we've been using LZMA. This will be rectified in the future, but should only be noticeable to users on old channels.</s> <span style="color:#009000">'''FIXED'''</span>: This is now taken care of, and several other paths were fixed as well.
+** We'll be sure to check the S3 metadata so that all filetypes in the cache can be downloaded properly, before final deployment.
+* '''Origin connections do not use TLS'''. When connecting to a Fastly POP, you use TLS. When Fastly POPs talk to each other, they also use TLS. When a POP talks to S3, '''the beta service does not use TLS''' -- it talks to S3 over HTTP. This is due to a limitation in a feature we use called '''[https://docs.fastly.com/guides/performance-tuning/streaming-miss Streaming Miss]''', which is vital in reducing <abbr title="Time To First Byte">TTFB</abbr> for large, uncached objects. (Without it, a POP must download an entire, possibly multi-hundred-MB NAR file before it can begin serving you. Streaming miss allows your download to start instantly.) Support for streaming miss with TLS origins is currently deployed in "Limited Availability" for Fastly customers. <s>We'll be applying to the LA program for TLS Origin support before deploying to production, and testing it carefully.</s> <span style="color:#009000">'''FIXED'''</span>: This is now taken care of -- the final, live deployment will use TLS Origins! The beta currently does not.
 == Known issues ==
@@ Line 70: / Line 83: @@
 === IPv6 shenanigans ===
-See [https://github.com/NixOS/nixpkgs/issues/65015]. Some users report that turning off IPv6 helps download things from <code>cache.nixos.org</code>. See also on the Fastly support forums: [https://support.fastly.com/hc/en-us/community/posts/360040169531-I-often-can-t-access-Fastly-servers-using-HTTPS-IPv6-RST-packets-received I (often) can't access Fastly servers using HTTPS+IPv6: RST packets received]. It is unclear how widespread this issue might be. Using an IPv4 only DNS CNAME may mitigate this in the long run.
+See [https://github.com/NixOS/nixpkgs/issues/65015]. Some users report that turning off IPv6 helps download things from <code>cache.nixos.org</code>. See also on the Fastly support forums: [https://web.archive.org/web/20221208122000/https://support.fastly.com/hc/en-us/community/posts/360040169531-I-often-can-t-access-Fastly-servers-using-HTTPS-IPv6-RST-packets-received I (often) can't access Fastly servers using HTTPS+IPv6: RST packets received]. It is unclear how widespread this issue might be. Using an IPv4 only DNS CNAME may mitigate this in the long run.
 == Future plans ==