暗无天日

=============>DarkSun的个人博客

curl练习

https://jvns.ca/blog/2019/08/27/curl-exercises/?utm_source=programmingdigest&utm_medium=web&utm_campaign=featured 上看到了一些 curl 的练习,觉得不错,尝试做一下,增强对 curl 的理解。

curl 的用法

curl.jpg

练习

  • Request https://httpbin.org

    curl https://httpbin.org |head
    
    <!DOCTYPE html>
    <html lang="en">
    
    <head>
        <meta charset="UTF-8">
        <title>httpbin.org</title>
        <link href="https://fonts.googleapis.com/css?family=Open+Sans:400,700|Source+Code+Pro:300,600|Titillium+Web:400,600,700"
            rel="stylesheet">
        <link rel="stylesheet" type="text/css" href="/flasgger_static/swagger-ui.css">
        <link rel="icon" type="image/png" href="/static/favicon.ico" sizes="64x64 32x32 16x16" />
    
  • Request https://httpbin.org/anything. httpbin.org/anything will look at the request you made, parse it, and echo back to you what you requested. curl’s default is to make a GET request.

    curl https://httpbin.org/anything
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "GET", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Make a POST request to https://httpbin.org/anything

    curl -X post https://httpbin.org/anything
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Content-Length": "0", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "POST", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Make a GET request to https://httpbin.org/anything, but this time add some query parameters (set value=panda).

    curl 'https://httpbin.org/anything?name=darksun&id=1'
    
    {
      "args": {
        "id": "1", 
        "name": "darksun"
      }, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "GET", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything?name=darksun&id=1"
    }
    
  • Request google’s robots.txt file (www.google.com/robots.txt)

    https_proxy=192.168.1.9:8118 curl https://www.google.com/robots.txt
    
    User-agent: *
    Disallow: /search
    Allow: /search/about
    Allow: /search/static
    Allow: /search/howsearchworks
    Disallow: /sdch
    Disallow: /groups
    Disallow: /index.html?
    Disallow: /?
    Allow: /?hl=
    Disallow: /?hl=*&
    Allow: /?hl=*&gws_rd=ssl$
    Disallow: /?hl=*&*&gws_rd=ssl
    Allow: /?gws_rd=ssl$
    Allow: /?pt1=true$
    Disallow: /imgres
    Disallow: /u/
    Disallow: /preferences
    Disallow: /setprefs
    Disallow: /default
    Disallow: /m?
    Disallow: /m/
    Allow:    /m/finance
    Disallow: /wml?
    Disallow: /wml/?
    Disallow: /wml/search?
    Disallow: /xhtml?
    Disallow: /xhtml/?
    Disallow: /xhtml/search?
    Disallow: /xml?
    Disallow: /imode?
    Disallow: /imode/?
    Disallow: /imode/search?
    Disallow: /jsky?
    Disallow: /jsky/?
    Disallow: /jsky/search?
    Disallow: /pda?
    Disallow: /pda/?
    Disallow: /pda/search?
    Disallow: /sprint_xhtml
    Disallow: /sprint_wml
    Disallow: /pqa
    Disallow: /palm
    Disallow: /gwt/
    Disallow: /purchases
    Disallow: /local?
    Disallow: /local_url
    Disallow: /shihui?
    Disallow: /shihui/
    Disallow: /products?
    Disallow: /product_
    Disallow: /products_
    Disallow: /products;
    Disallow: /print
    Disallow: /books/
    Disallow: /bkshp?*q=*
    Disallow: /books?*q=*
    Disallow: /books?*output=*
    Disallow: /books?*pg=*
    Disallow: /books?*jtp=*
    Disallow: /books?*jscmd=*
    Disallow: /books?*buy=*
    Disallow: /books?*zoom=*
    Allow: /books?*q=related:*
    Allow: /books?*q=editions:*
    Allow: /books?*q=subject:*
    Allow: /books/about
    Allow: /booksrightsholders
    Allow: /books?*zoom=1*
    Allow: /books?*zoom=5*
    Allow: /books/content?*zoom=1*
    Allow: /books/content?*zoom=5*
    Disallow: /ebooks/
    Disallow: /ebooks?*q=*
    Disallow: /ebooks?*output=*
    Disallow: /ebooks?*pg=*
    Disallow: /ebooks?*jscmd=*
    Disallow: /ebooks?*buy=*
    Disallow: /ebooks?*zoom=*
    Allow: /ebooks?*q=related:*
    Allow: /ebooks?*q=editions:*
    Allow: /ebooks?*q=subject:*
    Allow: /ebooks?*zoom=1*
    Allow: /ebooks?*zoom=5*
    Disallow: /patents?
    Disallow: /patents/download/
    Disallow: /patents/pdf/
    Disallow: /patents/related/
    Disallow: /scholar
    Disallow: /citations?
    Allow: /citations?user=
    Disallow: /citations?*cstart=
    Allow: /citations?view_op=new_profile
    Allow: /citations?view_op=top_venues
    Allow: /scholar_share
    Disallow: /s?
    Allow: /maps?*output=classic*
    Allow: /maps?*file=
    Allow: /maps/api/js?
    Allow: /maps/d/
    Disallow: /maps?
    Disallow: /mapstt?
    Disallow: /mapslt?
    Disallow: /maps/stk/
    Disallow: /maps/br?
    Disallow: /mapabcpoi?
    Disallow: /maphp?
    Disallow: /mapprint?
    Disallow: /maps/api/js/
    Disallow: /maps/api/place/js/
    Disallow: /maps/api/staticmap?
    Disallow: /maps/api/streetview?
    Disallow: /maps/api/streetview/
    Disallow: /maps/_/sw/manifest.json
    Disallow: /mld?
    Disallow: /staticmap?
    Disallow: /maps/preview
    Disallow: /maps/place
    Disallow: /maps/timeline/
    Disallow: /help/maps/streetview/partners/welcome/
    Disallow: /help/maps/indoormaps/partners/
    Disallow: /lochp?
    Disallow: /center
    Disallow: /ie?
    Disallow: /blogsearch/
    Disallow: /blogsearch_feeds
    Disallow: /advanced_blog_search
    Disallow: /uds/
    Disallow: /chart?
    Disallow: /transit?
    Allow:    /calendar$
    Allow:    /calendar/about/
    Disallow: /calendar/
    Disallow: /cl2/feeds/
    Disallow: /cl2/ical/
    Disallow: /coop/directory
    Disallow: /coop/manage
    Disallow: /trends?
    Disallow: /trends/music?
    Disallow: /trends/hottrends?
    Disallow: /trends/viz?
    Disallow: /trends/embed.js?
    Disallow: /trends/fetchComponent?
    Disallow: /trends/beta
    Disallow: /trends/topics
    Disallow: /musica
    Disallow: /musicad
    Disallow: /musicas
    Disallow: /musicl
    Disallow: /musics
    Disallow: /musicsearch
    Disallow: /musicsp
    Disallow: /musiclp
    Disallow: /urchin_test/
    Disallow: /movies?
    Disallow: /wapsearch?
    Allow: /safebrowsing/diagnostic
    Allow: /safebrowsing/report_badware/
    Allow: /safebrowsing/report_error/
    Allow: /safebrowsing/report_phish/
    Disallow: /reviews/search?
    Disallow: /orkut/albums
    Disallow: /cbk
    Allow: /cbk?output=tile&cb_client=maps_sv
    Disallow: /maps/api/js/AuthenticationService.Authenticate
    Disallow: /maps/api/js/QuotaService.RecordEvent
    Disallow: /recharge/dashboard/car
    Disallow: /recharge/dashboard/static/
    Disallow: /profiles/me
    Allow: /profiles
    Disallow: /s2/profiles/me
    Allow: /s2/profiles
    Allow: /s2/oz
    Allow: /s2/photos
    Allow: /s2/search/social
    Allow: /s2/static
    Disallow: /s2
    Disallow: /transconsole/portal/
    Disallow: /gcc/
    Disallow: /aclk
    Disallow: /cse?
    Disallow: /cse/home
    Disallow: /cse/panel
    Disallow: /cse/manage
    Disallow: /tbproxy/
    Disallow: /imesync/
    Disallow: /shenghuo/search?
    Disallow: /support/forum/search?
    Disallow: /reviews/polls/
    Disallow: /hosted/images/
    Disallow: /ppob/?
    Disallow: /ppob?
    Disallow: /accounts/ClientLogin
    Disallow: /accounts/ClientAuth
    Disallow: /accounts/o8
    Allow: /accounts/o8/id
    Disallow: /topicsearch?q=
    Disallow: /xfx7/
    Disallow: /squared/api
    Disallow: /squared/search
    Disallow: /squared/table
    Disallow: /qnasearch?
    Disallow: /app/updates
    Disallow: /sidewiki/entry/
    Disallow: /quality_form?
    Disallow: /labs/popgadget/search
    Disallow: /buzz/post
    Disallow: /compressiontest/
    Disallow: /analytics/feeds/
    Disallow: /analytics/partners/comments/
    Disallow: /analytics/portal/
    Disallow: /analytics/uploads/
    Allow: /alerts/manage
    Allow: /alerts/remove
    Disallow: /alerts/
    Allow: /alerts/$
    Disallow: /ads/search?
    Disallow: /ads/plan/action_plan?
    Disallow: /ads/plan/api/
    Disallow: /ads/hotels/partners
    Disallow: /phone/compare/?
    Disallow: /travel/clk
    Disallow: /travel/hotelier/terms/
    Disallow: /hotelfinder/rpc
    Disallow: /hotels/rpc
    Disallow: /commercesearch/services/
    Disallow: /evaluation/
    Disallow: /chrome/browser/mobile/tour
    Disallow: /compare/*/apply*
    Disallow: /forms/perks/
    Disallow: /shopping/suppliers/search
    Disallow: /ct/
    Disallow: /edu/cs4hs/
    Disallow: /trustedstores/s/
    Disallow: /trustedstores/tm2
    Disallow: /trustedstores/verify
    Disallow: /adwords/proposal
    Disallow: /shopping/product/
    Disallow: /shopping/seller
    Disallow: /shopping/ratings/account/metrics
    Disallow: /shopping/reviewer
    Disallow: /about/careers/applications/
    Disallow: /landing/signout.html
    Disallow: /webmasters/sitemaps/ping?
    Disallow: /ping?
    Disallow: /gallery/
    Disallow: /landing/now/ontap/
    Allow: /searchhistory/
    Allow: /maps/reserve
    Allow: /maps/reserve/partners
    Disallow: /maps/reserve/api/
    Disallow: /maps/reserve/search
    Disallow: /maps/reserve/bookings
    Disallow: /maps/reserve/settings
    Disallow: /maps/reserve/manage
    Disallow: /maps/reserve/payment
    Disallow: /maps/reserve/receipt
    Disallow: /maps/reserve/sellersignup
    Disallow: /maps/reserve/payments
    Disallow: /maps/reserve/feedback
    Disallow: /maps/reserve/terms
    Disallow: /maps/reserve/m/
    Disallow: /maps/reserve/b/
    Disallow: /maps/reserve/partner-dashboard
    Disallow: /about/views/
    Disallow: /intl/*/about/views/
    Disallow: /local/dining/
    Disallow: /local/place/products/
    Disallow: /local/place/reviews/
    Disallow: /local/place/rap/
    Disallow: /local/tab/
    Allow: /finance
    Allow: /js/
    Disallow: /finance?*q=*
    
    # AdsBot
    User-agent: AdsBot-Google
    Allow: /maps/api/js?
    Disallow: /maps/api/js/
    Disallow: /maps/api/place/js/
    Disallow: /maps/api/staticmap?
    Disallow: /maps/api/streetview?
    Disallow: /maps/api/streetview/
    
    # Certain social media sites are whitelisted to allow crawlers to access page markup when links to google.com/imgres* are shared. To learn more, please contact images-robots-whitelist@google.com.
    User-agent: Twitterbot
    Allow: /imgres
    
    User-agent: facebookexternalhit
    Allow: /imgres
    
    Sitemap: https://www.google.com/sitemap.xml
    
  • Make a GET request to https://httpbin.org/anything and set the header User-Agent: elephant.

    curl -H "User-Agent: elephant" 'https://httpbin.org/anything'
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Host": "httpbin.org", 
        "User-Agent": "elephant"
      }, 
      "json": null, 
      "method": "GET", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Make a DELETE request to https://httpbin.org/anything

    curl -X delete 'https://httpbin.org/anything'
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "DELETE", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Request https://httpbin.org/anything and also get the response headers

    curl -i 'https://httpbin.org/anything'
    
    HTTP/1.1 200 OK
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Origin: *
    Content-Type: application/json
    Date: Wed, 11 Sep 2019 07:43:57 GMT
    Referrer-Policy: no-referrer-when-downgrade
    Server: nginx
    X-Content-Type-Options: nosniff
    X-Frame-Options: DENY
    X-XSS-Protection: 1; mode=block
    Content-Length: 286
    Connection: keep-alive
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "GET", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Make a POST request to https://httpbin.com/anything with the JSON body {"value": "panda"}

    curl -X post 'https://httpbin.org/anything' --data '{"value":"panda"}'
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {
        "{\"value\":\"panda\"}": ""
      }, 
      "headers": {
        "Accept": "*/*", 
        "Content-Length": "17", 
        "Content-Type": "application/x-www-form-urlencoded", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "POST", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Make the same POST request as the previous exercise, but set the Content-Type header to application/json (because POST requests need to have a content type that matches their body). Look at the json field in the response to see the difference from the previous one.

    curl -X post -H 'Content-Type: application/json' 'https://httpbin.org/anything' --data '{"value":"panda"}'
    
    {
      "args": {}, 
      "data": "{\"value\":\"panda\"}", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Content-Length": "17", 
        "Content-Type": "application/json", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": {
        "value": "panda"
      }, 
      "method": "POST", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Make a GET request to https://httpbin.org/anything and set the header Accept-Encoding: gzip (what happens? why?)

    curl -X post -H 'Accept-Encoding: gzip' 'https://httpbin.org/anything' |gzip -d
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Accept-Encoding": "gzip", 
        "Content-Length": "0", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "POST", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Put a bunch of a JSON in a file and then make a POST request to https://httpbin.org/anything with the JSON in that file as the body

    tmp=$(mktemp)
    title="TEST OF JQ"
    source="JQ NB"
    echo '{}'|jq '{"title":$title,"content":$source}' --arg title "$title" --arg source "$source" >${tmp}
    curl -X post -H 'Content-Type: application/json'  'https://httpbin.org/anything' --data @${tmp}
    
    {
      "args": {}, 
      "data": "{  \"title\": \"TEST OF JQ\",  \"content\": \"JQ NB\"}", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Content-Length": "46", 
        "Content-Type": "application/json", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": {
        "content": "JQ NB", 
        "title": "TEST OF JQ"
      }, 
      "method": "POST", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Make a request to https://httpbin.org/image and set the header ‘Accept: image/png’. Save the output to a PNG file and open the file in an image viewer. Try the same thing with with different Accept: headers.

    png="images/curl_execise_1.png"
    curl -H "Accept: image/png" https://httpbin.org/image >${png}
    printf ${png}
    

    curl_execise_1.png

  • Make a PUT request to https://httpbin.org/anything

    curl -X put  https://httpbin.org/anything
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Content-Length": "0", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "PUT", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Request https://httpbin.org/image/jpeg, save it to a file, and open that file in your image editor.

    img="images/curl_execise_2.jpg"
    curl https://httpbin.org/image/jpeg >${img}
    printf ${img}
    

    curl_execise_2.jpg

  • Request https://www.twitter.com. You’ll get an empty response. Get curl to show you the response headers too, and try to figure out why the response was empty.

    https_proxy=192.168.1.9:8118 curl -i https://www.twitter.com
    
    HTTP/1.1 200 Connection established
    
    HTTP/2 301 
    content-length: 0
    date: Thu, 12 Sep 2019 12:51:19 GMT
    location: https://twitter.com/
    
    server: tsa_m
    set-cookie: personalization_id="v1_qj2JqquTZhCMoXb2fis0og=="; Max-Age=63072000; Expires=Sat, 11 Sep 2021 12:51:19 GMT; Path=/; Domain=.twitter.com
    set-cookie: guest_id=v1%3A156829267991786375; Max-Age=63072000; Expires=Sat, 11 Sep 2021 12:51:19 GMT; Path=/; Domain=.twitter.com
    strict-transport-security: max-age=631138519
    x-connection-hash: c6824d2630b6c4b240a7683f2f8e08d2
    x-response-time: 100
    
    
  • Make any request to https://httpbin.org/anything and just set some nonsense headers (like panda: elephant)

    curl -H "panda: elephant" https://httpbin.org/anything
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Host": "httpbin.org", 
        "Panda": "elephant", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "GET", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Request https://httpbin.org/status/404 and https://httpbin.org/status/200. Request them again and get curl to show the response headers.

    curl -i  https://httpbin.org/status/404
    curl -i  https://httpbin.org/status/200
    
    HTTP/1.1 404 NOT FOUND
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Origin: *
    Content-Type: text/html; charset=utf-8
    Date: Wed, 11 Sep 2019 11:20:41 GMT
    Referrer-Policy: no-referrer-when-downgrade
    Server: nginx
    X-Content-Type-Options: nosniff
    X-Frame-Options: DENY
    X-XSS-Protection: 1; mode=block
    Content-Length: 0
    Connection: keep-alive
    
    HTTP/1.1 200 OK
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Origin: *
    Content-Type: text/html; charset=utf-8
    Date: Wed, 11 Sep 2019 11:20:43 GMT
    Referrer-Policy: no-referrer-when-downgrade
    Server: nginx
    X-Content-Type-Options: nosniff
    X-Frame-Options: DENY
    X-XSS-Protection: 1; mode=block
    Content-Length: 0
    Connection: keep-alive
    
    
  • Request https://httpbin.org/anything and set a username and password (with -u username:password)

    curl -u lujun9972:1233 https://httpbin.org/anything
    
    {
      "args": {}, 
      "data": "", 
      "files": {}, 
      "form": {}, 
      "headers": {
        "Accept": "*/*", 
        "Authorization": "Basic bHVqdW45OTcyOjEyMzM=", 
        "Host": "httpbin.org", 
        "User-Agent": "curl/7.58.0"
      }, 
      "json": null, 
      "method": "GET", 
      "origin": "61.142.7.18, 61.142.7.18", 
      "url": "https://httpbin.org/anything"
    }
    
  • Download the Twitter homepage (https://twitter.com) in Spanish by setting the Accept-Language: es-ES header.

    https_proxy=192.168.1.9:8118 curl -H "Accept-Language: es-ES" https://twitter.com |tail
    
    
        <input type="hidden" class="swift-boot-module" value="app/pages/static_lohp">
      <input type="hidden" id="swift-module-path" value="https://abs.twimg.com/k/swift/es">
    
    
        <script src="https://abs.twimg.com/k/es/init.es.0df6a2c05184e170f1b7.js" async></script>
    
      </body>
    </html>