How To Extract All URLs From A Page Using PHP
0
Be First!
Recently I needed a crawler script that would create a list of all pages on a single domain. As a part of that I wrote some functions that could download a page, extract all URLs from the HTML and turn them into absolute URLs (so that they themselves can be crawled later). Here’s the PHP code.
Extracting All Links From A Page
Here’s a function that will download the specified URL and extract all links from the HTML. It also translates relative URLs to absolute URLs, tries to remove repeated links and is overall a fine piece of code 🙂 Depending on your goal you may want to comment out some lines (e.g. the part that strips ‘#something’ (in-page links) from URLs).
Here’s a function that will download the specified URL and extract all links from the HTML. It also translates relative URLs to absolute URLs, tries to remove repeated links and is overall a fine piece of code 🙂 Depending on your goal you may want to comment out some lines (e.g. the part that strips ‘#something’ (in-page links) from URLs).
Â
function crawl_page($page_url, $domain) {/* $page_url - page to extract links from, $domain -Â Â Â Â crawl only this domain (and subdomains)Â Â Â Â Returns an array of absolute URLs or false on failure.*/Â
/* I'm using cURL to retrieve the page */Â Â Â Â $ch = curl_init();Â Â Â Â curl_setopt($ch, CURLOPT_URL, $page_url);Â Â Â Â curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);Â Â Â Â curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);Â
/* Spoof the User-Agent header value; just to be safe */    curl_setopt($ch, CURLOPT_USERAGENT,      'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)');Â
/* I set timeout values for the connection and downloadbecause I don't want my script to get stuckdownloading huge files or trying to connect toa nonresponsive server. These are optional. */Â Â Â Â curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);Â Â Â Â curl_setopt($ch, CURLOPT_TIMEOUT, 15);Â
/* This ensures 404 Not Found (and similar) will be    treated as errors */    curl_setopt($ch, CURLOPT_FAILONERROR, true);Â
/* This might/should help against accidentally  downloading mp3 files and such, but it  doesn't really work :/ */    $header[] = "Accept: text/html, text/*";    curl_setopt($ch, CURLOPT_HTTPHEADER, $header);Â
/* Download the page */Â Â Â Â $html = curl_exec($ch);Â Â Â Â curl_close($ch);Â
    if(!$html) return false;Â
/* Extract the BASE tag (if present) for  relative-to-absolute URL conversions later */    if(preg_match('/]+)[\'\" >]/i' ,$html, $matches)){        $base_url=$matches[1];    } else {        $base_url=$page_url;    }Â
    $links=array();Â
    $html = str_replace("\n", ' ', $html);    preg_match_all('/]*href\s*=\s*([\"\']+)([^>]+?)(\1|>)/i' , $html, $m);/* this regexp is a combination of numerous    versions I saw online; should be good. */Â
    foreach($m[2] as $url) {        $url=trim($url);Â
        /* get rid of PHPSESSID, #linkname, & and javascript: */        $url=preg_replace(            array('/([\?&]PHPSESSID=\w+)$/i','/(#[^\/]*)$/i', '/&/','/^(javascript:.*)/i'),            array('','','&',''),            $url);Â
        /* turn relative URLs into absolute URLs.          relative2absolute() is defined further down          below on this page. */            $url = relative2absolute($base_url, $url);   Â
            // check if in the same (sub-)$domain            if(preg_match("/^http[s]?:\/\/[^\/]*".str_replace('.', '\.', $domain)."/i", $url)) {                //save the URL                if(!in_array($url, $links)) $links[]=$url;            }    }Â
    return $links;}How To Translate a Relative URL to an Absolute URL
This script is based on a function I found on the web with some small but significant changes.
function relative2absolute($absolute, $relative) {
        $p = @parse_url($relative);
        if(!$p) {
            //$relative is a seriously malformed URL
            return false;
        }
        if(isset($p["scheme"])) return $relative;
Â
        $parts=(parse_url($absolute));
Â
        if(substr($relative,0,1)=='/') {
            $cparts = (explode("/", $relative));
            array_shift($cparts);
        } else {
            if(isset($parts['path'])){
                 $aparts=explode('/',$parts['path']);
                 array_pop($aparts);
                 $aparts=array_filter($aparts);
            } else {
                 $aparts=array();
            }
           $rparts = (explode("/", $relative));
           $cparts = array_merge($aparts, $rparts);
           foreach($cparts as $i => $part) {
                if($part == '.') {
                    unset($cparts[$i]);
                } else if($part == '..') {
                    unset($cparts[$i]);
                    unset($cparts[$i-1]);
                }
            }
        }
        $path = implode("/", $cparts);
Â
        $url = '';
        if($parts['scheme']) {
            $url = "$parts[scheme]://";
        }
        if(isset($parts['user'])) {
            $url .= $parts['user'];
            if(isset($parts['pass'])) {
                $url .= ":".$parts['pass'];
            }
            $url .= "@";
        }
        if(isset($parts['host'])) {
            $url .= $parts['host']."/";
        }
        $url .= $path;
Â
        return $url;
}
(479)
