Home Uncategorized How To Extract All URLs From A Page Using PHP

Recently I needed a crawler script that would create a list of all pages on a single domain. As a part of that I wrote some functions that could download a page, extract all URLs from the HTML and turn them into absolute URLs (so that they themselves can be crawled later). Here’s the PHP code.

Extracting All Links From A Page
Here’s a function that will download the specified URL and extract all links from the HTML. It also translates relative URLs to absolute URLs, tries to remove repeated links and is overall a fine piece of code 🙂 Depending on your goal you may want to comment out some lines (e.g. the part that strips ‘#something’ (in-page links) from URLs).
function crawl_page($page_url, $domain) {
/* $page_url - page to extract links from, $domain -
    crawl only this domain (and subdomains)
    Returns an array of absolute URLs or false on failure.
/* I'm using cURL to retrieve the page */
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $page_url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
/* Spoof the User-Agent header value; just to be safe */
    curl_setopt($ch, CURLOPT_USERAGENT,
      'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)');
/* I set timeout values for the connection and download
because I don't want my script to get stuck
downloading huge files or trying to connect to
a nonresponsive server. These are optional. */
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
    curl_setopt($ch, CURLOPT_TIMEOUT, 15);
/* This ensures 404 Not Found (and similar) will be
    treated as errors */
    curl_setopt($ch, CURLOPT_FAILONERROR, true);
/* This might/should help against accidentally
  downloading mp3 files and such, but it
  doesn't really work :/  */
    $header[] = "Accept: text/html, text/*";
    curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
/* Download the page */
    $html = curl_exec($ch);
    if(!$html) return false;
/* Extract the BASE tag (if present) for
  relative-to-absolute URL conversions later */
    if(preg_match('/]+)[\'\" >]/i',$html, $matches)){
    } else {
    $html = str_replace("\n", ' ', $html);
    preg_match_all('/]*href\s*=\s*([\"\']+)([^>]+?)(\1|>)/i', $html, $m);
/* this regexp is a combination of numerous
    versions I saw online; should be good. */
    foreach($m[2] as $url) {
        /* get rid of PHPSESSID, #linkname, & and javascript: */
            array('/([\?&]PHPSESSID=\w+)$/i','/(#[^\/]*)$/i', '/&/','/^(javascript:.*)/i'),
        /* turn relative URLs into absolute URLs.
          relative2absolute() is defined further down
          below on this page. */
            $url = relative2absolute($base_url, $url);   
            // check if in the same (sub-)$domain
            if(preg_match("/^http[s]?:\/\/[^\/]*".str_replace('.', '\.', $domain)."/i", $url)) {
                //save the URL
                if(!in_array($url, $links)) $links[]=$url;
    return $links;
How To Translate a Relative URL to an Absolute URL
This script is based on a function I found on the web with some small but significant changes.

function relative2absolute($absolute, $relative) {
        $p = @parse_url($relative);
        if(!$p) {
            //$relative is a seriously malformed URL
            return false;
        if(isset($p["scheme"])) return $relative;
        if(substr($relative,0,1)=='/') {
            $cparts = (explode("/", $relative));
        } else {
            } else {
           $rparts = (explode("/", $relative));
           $cparts = array_merge($aparts, $rparts);
           foreach($cparts as $i => $part) {
                if($part == '.') {
                } else if($part == '..') {
        $path = implode("/", $cparts);
        $url = '';
        if($parts['scheme']) {
            $url = "$parts[scheme]://";
        if(isset($parts['user'])) {
            $url .= $parts['user'];
            if(isset($parts['pass'])) {
                $url .= ":".$parts['pass'];
            $url .= "@";
        if(isset($parts['host'])) {
            $url .= $parts['host']."/";
        $url .= $path;
        return $url;

You may also like

Leave a Comment