Skip to content

Core Module

This module provides core requests_whaor functionality.

pause

def pause(sleep: int) -> None

Sleep function with a little logging fun.

Requestor

Makes proxied web requests via a rotating proxy TOR network.

__init__

 | def __init__(onions: List[OnionCircuit], onion_balancer: Balancer, timeout: int, max_retries: int) -> "Requestor"

Requestor init method.

Arguments:

  • onions List[OnionCircuit] - List of TOR containers.
  • onion_balancer Balancer - Balancer instances connected to TOR containers on the same network.
  • timeout int - Requests timeout.
  • max_retries int - Max number of time to retry on bad response or connection error.

rotating_proxy

 | def rotating_proxy() -> Dict[str, str]

Rotating proxy frontend input address.

get

 | def get(url: str, *args, **kwargs) -> Optional[requests.models.Response]

Overload requests.get method.

This will pass in the rotating proxy host address and timeout into the requests.get method. Additionally, It provides a way to automatically retry on connection failures and bad status_codes. Each time there is a failure it will try a new request with a new ip address.

Arguments:

  • url str - url to send the get request.
  • *args - arguments to pass to requests.get() method.
  • **kwargs - keyword arguments to pass to requests.get() method.

Returns:

  • Response - If a response is found else None.

restart_onions

 | def restart_onions(with_threads: bool = True, max_threads: int = 5) -> None

Restart onion containers.

This can be useful for changing ip addresses every n requests.

Arguments:

  • with_threads bool - if True uses threads to restart the containers.
  • max_threads int - How many threads to use.

RequestsWhaor

def RequestsWhaor(onion_count: int = 5, start_with_threads: bool = True, max_threads: int = 5, timeout: int = 5, show_log: bool = False, max_retries: int = 5) -> Requestor

Context manager which starts n amount of tor nodes behind a round robin reverse proxy.

Arguments:

  • onion_count int - Number of TOR circuits to spin up.
  • start_with_threads bool - If True uses treads to spin up containers.
  • max_threads int - Max number of threads to use when spin up containers.
  • timeout int - Requests timeout.
  • show_log bool - If True shows the containers logs.
  • max_retries int - Max number of time to retry on bad response or connection error.

Yields:

  • Requestor - Makes proxied web requests via a rotating proxy TOR network.