Skip to content

Commit 3921e2a

Browse files
committed
doc updates
1 parent 643e247 commit 3921e2a

File tree

1 file changed

+11
-5
lines changed

1 file changed

+11
-5
lines changed

README.md

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,30 @@
1-
The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests.
1+
# Scrapy Proxy Headers
2+
3+
The `scrapy-proxy-headers` package is designed for adding proxy headers to HTTPS requests in [Scrapy](https://scrapy.org/).
24

35
In normal usage, custom headers put in `request.headers` cannot be read by a proxy when you make a HTTPS request, because the headers are encrypted and passed through the proxy tunnel, along with the rest of the request body. You can read more about this at [Proxy Server Requests over HTTPS](https://docs.proxymesh.com/article/145-proxy-server-requests-over-https).
46

5-
Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for any custom headers through a proxy.
7+
Because Scrapy does not have a good way to pass custom headers to a proxy when you make HTTPS requests, we at [ProxyMesh](https://proxymesh.com) made this extension to support our customers that use Scrapy and want to use custom headers to control our proxy behavior. But this extension can work for handling custom headers through any proxy.
8+
9+
## Installation
610

711
To use this extension, do the following:
812

913
1. `pip install scrapy-proxy-headers`
10-
2. In your Scrapy `settings.py`, add the following code:
14+
2. In your Scrapy `settings.py`, add the following:
1115

1216
```python
1317
DOWNLOAD_HANDLERS = {
1418
"https": "scrapy_proxy_headers.HTTP11ProxyDownloadHandler"
1519
}
1620
```
1721

18-
3. When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
22+
## Usage
23+
24+
When you want make a request with a custom proxy header, instead of using `request.headers`, use `request.meta["proxy_headers"]` like this:
1925

2026
```python
2127
request.meta["proxy_headers"] = {"X-ProxyMesh-Country": "US"}
2228
```
2329

24-
4. Any response headers that might come from the proxy will be saved in `response.headers`
30+
Any response headers that might come from the proxy will be saved in `response.headers`, as in `response.headers["X-ProxyMesh-IP"]`.

0 commit comments

Comments
 (0)