☰

我怎麼能curl -IL從根頁面開始的所有網站頁面而不僅僅是1頁？

tag-icon • tag-icon nginx cache fastcgi curl
我怎麼能curl -IL從根頁面開始的所有網站頁面而不僅僅是1頁？

我需要curl -IL所有網站，也許每24小時做一次cron工作或其他事情，這樣fastcgi就可以快取所有頁面維戈

除非我捲曲 -IL 網頁，否則快取不會命中，因此我需要一些可以捲曲網站上所有可用網頁的東西。

就像你可以看到下面這樣：

root@ubuntu-s-1vcpu-1gb-amd-sfo3-01:~# curl -IL https://www.example.com
HTTP/2 200 
date: Thu, 23 Feb 2023 10:31:44 GMT
content-type: text/html; charset=UTF-8
link: <https://www.example.com/wp-json/>; rel="https://api.w.org/"
link: <https://www.example.com/wp-json/wp/v2/pages/16274>; rel="alternate"; type="application/json"
link: <https://www.example.com/>; rel=shortlink
x-fastcgi-cache: HIT
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2K%2BxMLMMrvGpe%2FdpG38gU%2FKaBPBAL5I3rYNL5ATQ65wPM2qMnQmaCe2jLx88N%2B%2BHeUw1RRI5EnrYh9%2F6P5fj7xAipg%2FnOxi4IV8e4IqPWMyANW9JnKndXGfBZw0r3LtF1IEl"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 79df4b241af32f15-LAX
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

答案1

這就是您在那裡使用的 WordPress。我確信有更好的方法來預熱快取：https://de.wordpress.org/plugins/search/cache+warm/

使用以下程式碼建立一個類似warmup.sh的文件

#!/bin/bash

# URL of main Sitemap
sitemap_url="https://www.vgopromo.com/wp-sitemap.xml"

# Extract all Sitemap URLs
sitemap_urls=$(curl -s "$sitemap_url" | grep -oP '(?<=<loc>)[^<]+')

# Loop over and retrieve the individual URLs
for sitemap in $sitemap_urls; do
    urls=$(curl -s "$sitemap" | grep -oP '(?<=<loc>)[^<]+')
    for url in $urls; do
        curl -IL "$url"
    done
done

這將執行您所要求的操作。

您也可以使用 Cronjob 來運行此文件。

# Example: At minute 15 past every hour.
15 */1 * * * /bin/bash /root/warmup.sh

編輯

此修改後的程式碼新增了也定義子網域的選項。

#!/bin/bash

# Array of Subdomains, just extend in same princip
subdomains=("www" "subdomain_2")

# Loop over Subdomains and retrieve URLs
for subdomain in "${subdomains[@]}"; do
    sitemap_url="https://$subdomain.vgopromo.com/wp-sitemap.xml"
    sitemap_urls=$(curl -s "$sitemap_url" | grep -oP '(?<=<loc>)[^<]+')
    for sitemap in $sitemap_urls; do
        urls=$(curl -s "$sitemap" | grep -oP '(?<=<loc>)[^<]+')
        for url in $urls; do
            curl -IL "$url"
        done
    done
done

個人建議採用這種方式。使其與多個項目相容

#!/bin/bash

# Array of Domains, just extend in same princip
domains=("https://www.vgopromo.com/wp-sitemap.xml" "https://example.vgopromo.com/wp-sitemap.xml")

# Loop over Domains and retrieve URLs
for domain in "${domains[@]}"; do
    sitemap_url="$domain"
    sitemap_urls=$(curl -s "$sitemap_url" | grep -oP '(?<=<loc>)[^<]+')
    for sitemap in $sitemap_urls; do
        urls=$(curl -s "$sitemap" | grep -oP '(?<=<loc>)[^<]+')
        for url in $urls; do
            curl -IL "$url"
        done
    done
done

相關內容