Online: http://curl.haxx.se/docs/httpscripting.shtml
-Date: December 7, 2004
+Date: December 9, 2004
The Art Of Scripting HTTP Requests Using Curl
=============================================
you need to replace that space with %20 etc. Failing to comply with this
will most likely cause your data to be received wrongly and messed up.
- 4.3 FILE UPLOAD POST
+ 4.3 File Upload POST
- Back in late 1995 they defined a new way to post data over HTTP. It was
- documented in the RFC 1867, why this method sometimes is referred to as
- a RFC1867-posting.
+ Back in late 1995 they defined an additional way to post data over HTTP. It
+ is documented in the RFC 1867, why this method sometimes is referred to as
+ RFC1867-posting.
This method is mainly designed to better support file uploads. A form that
allows a user to upload a file could be written like this in HTML:
curl -F upload=@localfilename -F press=OK [URL]
- 4.4 HIDDEN FIELDS
+ 4.4 Hidden Fields
A very common way for HTML based application to pass state information
between pages is to add hidden fields to the forms. Hidden fields are
curl -d "birthyear=1905&press=OK&person=daniel" [URL]
- 4.5 FIGURE OUT WHAT A POST LOOKS LIKE
+ 4.5 Figure Out What A POST Looks Like
When you're about fill in a form and send to a server by using curl instead
of a browser, you're of course very interested in sending a POST exactly the
curl -T uploadfile www.uploadhttp.com/receive.cgi
-6. AUTHENTICATION
+6. Authentication
Authentication is the ability to tell the server your username and password
so that it can verify that you're allowed to do the request you're doing. The
able to watch your passwords if you pass them as plain command line
options. There are ways to circumvent this.
-7. REFERER
+7. Referer
A HTTP request may include a 'referer' field (yes it is misspelled), which
can be used to tell from which URL the client got to this particular
curl -e http://curl.haxx.se daniel.haxx.se
-8. USER AGENT
+8. User Agent
Very similar to the referer field, all HTTP requests may set the User-Agent
field. It names what user agent (client) that is being used. Many
curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
-9. REDIRECTS
+9. Redirects
When a resource is requested from a server, the reply from the server may
include a hint about where the browser should go next to find this page, or a
page, you can safely use -L and -d/-F together. Curl will only use POST in
the first request, and then revert to GET in the following operations.
-10. COOKIES
+10. Cookies
The way the web browsers do "client side state control" is by using
cookies. Cookies are just names with associated contents. The cookies are
curl https://that.secure.server.com
- 11.1 CERTIFICATES
+ 11.1 Certificates
In the HTTPS world, you use certificates to validate that you are the one
you you claim to be, as an addition to normal passwords. Curl supports
http://curl.haxx.se/docs/sslcerts.html
-12. REFERENCES
+12. Custom Request Elements
+
+ Doing fancy stuff, you may need to add or change elements of a single curl
+ request.
+
+ For example, you can change the POST request to a PROPFIND and send the data
+ as "Content-Type: text/xml" (instead of the default Content-Type) like this:
+
+ curl -d "<xml>" -H "Content-Type: text/xml" -X PROPFIND url.com
+
+ You can delete a default header by providing one without content. Like you
+ can ruin the request by chopping off the Host: header:
+
+ curl -H "Host:" http://mysite.com
+
+ You can add headers the same way. Your server may want a "Destination:"
+ header, and you can add it:
+
+ curl -H "Destination: http://moo.com/nowhere" http://url.com
+
+13. Debug
+
+ Many times when you run curl on a site, you'll notice that the site doesn't
+ seem to respond the same way to your curl requests as it does to your
+ browser's.
+
+ Then you need to start making your curl requests more similar to your
+ browser's requests:
+
+ * Use the --trace-ascii option to store fully detailed logs of the requests
+ for easier analyzing and better understanding
+
+ * Make sure you check for and use cookies when needed (both reading with -b
+ and writing with -c)
+
+ * Set user-agent to one like a recent popular browser does
+
+ * Set referer like it is set by the browser
+
+ * If you use POST, make sure you send all the fields and in the same order as
+ the browser does it. (See chapter 4.5 above)
+
+ A very good helper to make sure you do this right, is the LiveHTTPHeader tool
+ that lets you view all headers you send and receive with Mozilla/Firefox
+ (even when using HTTPS).
+
+ A more raw approach is to capture the HTTP traffic on the network with tools
+ such as ethereal or tcpdump and check what headers that were sent and
+ received by the browser. (HTTPS makes this technique inefficient.)
+
+14. References
RFC 2616 is a must to read if you want in-depth understanding of the HTTP
protocol.
projects / thirdparty / curl.git / commitdiff
