Core Module
This module provides core requests_whaor functionality.
pause
def pause(sleep: int) -> None
Sleep function with a little logging fun.
Requestor
Makes proxied web requests via a rotating proxy TOR network.
__init__
| def __init__(onions: List[OnionCircuit], onion_balancer: Balancer, timeout: int, max_retries: int) -> "Requestor"
Requestor init method.
Arguments:
onions
List[OnionCircuit] - List of TOR containers.onion_balancer
Balancer - Balancer instances connected to TOR containers on the same network.timeout
int - Requests timeout.max_retries
int - Max number of time to retry on bad response or connection error.
rotating_proxy
| def rotating_proxy() -> Dict[str, str]
Rotating proxy frontend input address.
get
| def get(url: str, *args, **kwargs) -> Optional[requests.models.Response]
Overload requests.get method.
This will pass in the rotating proxy host address and timeout into the requests.get method. Additionally, It provides a way to automatically retry on connection failures and bad status_codes. Each time there is a failure it will try a new request with a new ip address.
Arguments:
url
str - url to send the get request.*args
- arguments to pass to requests.get() method.**kwargs
- keyword arguments to pass to requests.get() method.Returns:
Response
- If a response is found else None.
restart_onions
| def restart_onions(with_threads: bool = True, max_threads: int = 5) -> None
Restart onion containers.
This can be useful for changing ip addresses every n requests.
Arguments:
with_threads
bool - if True uses threads to restart the containers.max_threads
int - How many threads to use.
RequestsWhaor
def RequestsWhaor(onion_count: int = 5, start_with_threads: bool = True, max_threads: int = 5, timeout: int = 5, show_log: bool = False, max_retries: int = 5) -> Requestor
Context manager which starts n amount of tor nodes behind a round robin reverse proxy.
Arguments:
onion_count
int - Number of TOR circuits to spin up.start_with_threads
bool - If True uses treads to spin up containers.max_threads
int - Max number of threads to use when spin up containers.timeout
int - Requests timeout.show_log
bool - If True shows the containers logs.max_retries
int - Max number of time to retry on bad response or connection error.Yields:
Requestor
- Makes proxied web requests via a rotating proxy TOR network.