Puppeteer Proxy Setup and Configuration Guide

Puppeteer is a Node.js library developed by Google that allows programmatic control of the Chromium browser in headless (invisible) or headed (visible) mode. Puppeteer's key advantage in web scraping is its JavaScript rendering capability: modern websites built with frameworks like React, Vue, and Angular load their content with JavaScript, and you cannot access this content with simple HTTP requests. Puppeteer loads the page like a real browser, executes JavaScript, and renders the DOM. When used with a proxy, you can perform this operation from different IP addresses, bypassing rate-limit and IP ban issues.

There are several methods for proxy authentication in Puppeteer. The most common is the page.authenticate() method: you call await page.authenticate({ username: "user", password: "pass" }) each time you create a new page. This method automatically sends Chromium's HTTP Proxy-Authorization header. The second method is using Chrome DevTools Protocol (CDP): create a CDP session with page.target().createCDPSession() and manage proxy auth credentials with Fetch.enable. The third method is IP authorization: define your server IP in the ProxyTurk panel and you don't need to send auth credentials. IP authorization is the most practical method for server-based scraping.

Modern websites use various techniques to detect headless browsers. The most common is checking if the navigator.webdriver property is true — Puppeteer sets this value to true by default. The puppeteer-extra-plugin-stealth plugin automatically hides this and many other detection points: WebGL renderer, Chrome runtime, permissions API, and more. Additionally, use realistic viewport sizes (like 1920x1080, 1366x768), set the Accept-Language header appropriate for the target country, and configure the timezone according to the proxy location. Simulating real user behaviors like mouse movements and page scrolling also reduces detection risk.

When scraping with Puppeteer, resource consumption is high since each page load is a real browser session. To optimize performance, first block unnecessary resource loading: use page.setRequestInterception(true) to abort image, font, CSS, and ad requests — this both increases speed and saves bandwidth. Reuse browser instances: open multiple pages on a single browser instead of opening a new browser for each request. Use libraries like p-limit for parallel page management to control concurrent tab count (typically 5-10 is optimal). Use page.waitForSelector() to wait only for required elements, and avoid unnecessary waits with page.waitForTimeout().