tornado.httpclient — Asynchronous HTTP client
Blocking and non-blocking HTTP client interfaces.
This module defines a common interface shared by two implementations,
simple_httpclient and curl_httpclient. Applications may either
instantiate their chosen implementation class directly or use the
AsyncHTTPClient class from this module, which selects an implementation
that can be overridden with the AsyncHTTPClient.configure method.
The default implementation is simple_httpclient, and this is expected
to be suitable for most users’ needs. However, some applications may wish
to switch to curl_httpclient for reasons such as the following:
- curl_httpclient has some features not found in simple_httpclient,
including support for HTTP proxies and the ability to use a specified
network interface.
- curl_httpclient is more likely to be compatible with sites that are
not-quite-compliant with the HTTP spec, or sites that use little-exercised
features of HTTP.
- curl_httpclient is faster.
- curl_httpclient was the default prior to Tornado 2.0.
Note that if you are using curl_httpclient, it is highly recommended that
you use a recent version of libcurl and pycurl. Currently the minimum
supported version is 7.18.2, and the recommended version is 7.21.1 or newer.
It is highly recommended that your libcurl installation is built with
asynchronous DNS resolver (threaded or c-ares), otherwise you may encounter
various problems with request timeouts (for more information, see
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTCONNECTTIMEOUTMS
and comments in curl_httpclient.py).
HTTP client interfaces
-
class tornado.httpclient.HTTPClient(async_client_class=None, **kwargs)[source]
A blocking HTTP client.
This interface is provided for convenience and testing; most applications
that are running an IOLoop will want to use AsyncHTTPClient instead.
Typical usage looks like this:
http_client = httpclient.HTTPClient()
try:
response = http_client.fetch("http://www.google.com/")
print response.body
except httpclient.HTTPError as e:
print "Error:", e
http_client.close()
-
close()[source]
Closes the HTTPClient, freeing any resources used.
-
fetch(request, **kwargs)[source]
Executes a request, returning an HTTPResponse.
The request may be either a string URL or an HTTPRequest object.
If it is a string, we construct an HTTPRequest using any additional
kwargs: HTTPRequest(request, **kwargs)
If an error occurs during the fetch, we raise an HTTPError.
-
class tornado.httpclient.AsyncHTTPClient[source]
An non-blocking HTTP client.
Example usage:
def handle_request(response):
if response.error:
print "Error:", response.error
else:
print response.body
http_client = AsyncHTTPClient()
http_client.fetch("http://www.google.com/", handle_request)
The constructor for this class is magic in several respects: It
actually creates an instance of an implementation-specific
subclass, and instances are reused as a kind of pseudo-singleton
(one per IOLoop). The keyword argument force_instance=True
can be used to suppress this singleton behavior. Constructor
arguments other than io_loop and force_instance are
deprecated. The implementation subclass as well as arguments to
its constructor can be set with the static method configure()
-
close()[source]
Destroys this HTTP client, freeing any file descriptors used.
This method is not needed in normal use due to the way
that AsyncHTTPClient objects are transparently reused.
close() is generally only necessary when either the
IOLoop is also being closed, or the force_instance=True
argument was used when creating the AsyncHTTPClient.
No other methods may be called on the AsyncHTTPClient after
close().
-
fetch(request, callback=None, **kwargs)[source]
Executes a request, asynchronously returning an HTTPResponse.
The request may be either a string URL or an HTTPRequest object.
If it is a string, we construct an HTTPRequest using any additional
kwargs: HTTPRequest(request, **kwargs)
This method returns a Future whose result is an
HTTPResponse. The Future will raise an HTTPError if
the request returned a non-200 response code.
If a callback is given, it will be invoked with the HTTPResponse.
In the callback interface, HTTPError is not automatically raised.
Instead, you must check the response’s error attribute or
call its rethrow method.
-
classmethod configure(impl, **kwargs)[source]
Configures the AsyncHTTPClient subclass to use.
AsyncHTTPClient() actually creates an instance of a subclass.
This method may be called with either a class object or the
fully-qualified name of such a class (or None to use the default,
SimpleAsyncHTTPClient)
If additional keyword arguments are given, they will be passed
to the constructor of each subclass instance created. The
keyword argument max_clients determines the maximum number
of simultaneous fetch() operations that can
execute in parallel on each IOLoop. Additional arguments
may be supported depending on the implementation class in use.
Example:
AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
Request objects
-
class tornado.httpclient.HTTPRequest(url, method='GET', headers=None, body=None, auth_username=None, auth_password=None, auth_mode=None, connect_timeout=None, request_timeout=None, if_modified_since=None, follow_redirects=None, max_redirects=None, user_agent=None, use_gzip=None, network_interface=None, streaming_callback=None, header_callback=None, prepare_curl_callback=None, proxy_host=None, proxy_port=None, proxy_username=None, proxy_password=None, allow_nonstandard_methods=None, validate_cert=None, ca_certs=None, allow_ipv6=None, client_key=None, client_cert=None, body_producer=None, expect_100_continue=False)[source]
HTTP client request object.
All parameters except url are optional.
Parameters: |
- url (string) – URL to fetch
- method (string) – HTTP method, e.g. “GET” or “POST”
- headers (HTTPHeaders or dict) – Additional HTTP headers to pass on the request
- body – HTTP request body as a string (byte or unicode; if unicode
the utf-8 encoding will be used)
- body_producer – Callable used for lazy/asynchronous request bodies.
It is called with one argument, a write function, and should
return a Future. It should call the write function with new
data as it becomes available. The write function returns a
Future which can be used for flow control.
Only one of body and body_producer may
be specified. body_producer is not supported on
curl_httpclient. When using body_producer it is recommended
to pass a Content-Length in the headers as otherwise chunked
encoding will be used, and many servers do not support chunked
encoding on requests. New in Tornado 3.3
- auth_username (string) – Username for HTTP authentication
- auth_password (string) – Password for HTTP authentication
- auth_mode (string) – Authentication mode; default is “basic”.
Allowed values are implementation-defined; curl_httpclient
supports “basic” and “digest”; simple_httpclient only supports
“basic”
- connect_timeout (float) – Timeout for initial connection in seconds
- request_timeout (float) – Timeout for entire request in seconds
- if_modified_since (datetime or float) – Timestamp for If-Modified-Since header
- follow_redirects (bool) – Should redirects be followed automatically
or return the 3xx response?
- max_redirects (int) – Limit for follow_redirects
- user_agent (string) – String to send as User-Agent header
- use_gzip (bool) – Request gzip encoding from the server
- network_interface (string) – Network interface to use for request.
curl_httpclient only; see note below.
- streaming_callback (callable) – If set, streaming_callback will
be run with each chunk of data as it is received, and
HTTPResponse.body and HTTPResponse.buffer will be empty in
the final response.
- header_callback (callable) – If set, header_callback will
be run with each header line as it is received (including the
first line, e.g. HTTP/1.0 200 OK\r\n, and a final line
containing only \r\n. All lines include the trailing newline
characters). HTTPResponse.headers will be empty in the final
response. This is most useful in conjunction with
streaming_callback, because it’s the only way to get access to
header data while the request is in progress.
- prepare_curl_callback (callable) – If set, will be called with
a pycurl.Curl object to allow the application to make additional
setopt calls.
- proxy_host (string) – HTTP proxy hostname. To use proxies,
proxy_host and proxy_port must be set; proxy_username and
proxy_pass are optional. Proxies are currently only supported
with curl_httpclient.
- proxy_port (int) – HTTP proxy port
- proxy_username (string) – HTTP proxy username
- proxy_password (string) – HTTP proxy password
- allow_nonstandard_methods (bool) – Allow unknown values for method
argument?
- validate_cert (bool) – For HTTPS requests, validate the server’s
certificate?
- ca_certs (string) – filename of CA certificates in PEM format,
or None to use defaults. See note below when used with
curl_httpclient.
- allow_ipv6 (bool) – Use IPv6 when available? Default is false in
simple_httpclient and true in curl_httpclient
- client_key (string) – Filename for client SSL key, if any. See
note below when used with curl_httpclient.
- client_cert (string) – Filename for client SSL certificate, if any.
See note below when used with curl_httpclient.
- expect_100_continue (bool) – If true, send the
Expect: 100-continue header and wait for a continue response
before sending the request body. Only supported with
simple_httpclient.
|
Note
When using curl_httpclient certain options may be
inherited by subsequent fetches because pycurl does
not allow them to be cleanly reset. This applies to the
ca_certs, client_key, client_cert, and
network_interface arguments. If you use these
options, you should pass them on every request (you don’t
have to always use the same values, but it’s not possible
to mix requests that specify these options with ones that
use the defaults).
New in version 3.1: The auth_mode argument.
New in version 3.3: The body_producer and expect_100_continue arguments.
Response objects
-
class tornado.httpclient.HTTPResponse(request, code, headers=None, buffer=None, effective_url=None, error=None, request_time=None, time_info=None, reason=None)[source]
HTTP Response object.
Attributes:
- request: HTTPRequest object
- code: numeric HTTP status code, e.g. 200 or 404
- reason: human-readable reason phrase describing the status code
(with curl_httpclient, this is a default value rather than the
server’s actual response)
- headers: tornado.httputil.HTTPHeaders object
- effective_url: final location of the resource after following any
redirects
- buffer: cStringIO object for response body
- body: response body as string (created on demand from self.buffer)
- error: Exception object, if any
- request_time: seconds from request start to finish
- time_info: dictionary of diagnostic timing information from the request.
Available data are subject to change, but currently uses timings
available from http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html,
plus queue, which is the delay (if any) introduced by waiting for
a slot under AsyncHTTPClient‘s max_clients setting.
-
rethrow()[source]
If there was an error on the request, raise an HTTPError.
Exceptions
-
exception tornado.httpclient.HTTPError(code, message=None, response=None)[source]
Exception thrown for an unsuccessful HTTP request.
Attributes:
- code - HTTP error integer error code, e.g. 404. Error code 599 is
used when no HTTP response was received, e.g. for a timeout.
- response - HTTPResponse object, if any.
Note that if follow_redirects is False, redirects become HTTPErrors,
and you can look at error.response.headers['Location'] to see the
destination of the redirect.
Command-line interface
This module provides a simple command-line interface to fetch a url
using Tornado’s HTTP client. Example usage:
# Fetch the url and print its body
python -m tornado.httpclient http://www.google.com
# Just print the headers
python -m tornado.httpclient --print_headers --print_body=false http://www.google.com