westworld: a webmasters best friend » scrape validate http://www.westworld.be A webmasters best friend Mon, 14 Jun 2010 18:12:24 +0000 http://wordpress.org/?v=2.8.6 en hourly 1 scraping the web http://www.westworld.be/a-note-to-self/wget-as-a-spider/ http://www.westworld.be/a-note-to-self/wget-as-a-spider/#comments Wed, 07 Jan 2009 08:22:19 +0000 westworld http://www.westworld.be/?p=65
  • Prototype: submit multiple select with ajax call
    • ]]>     I’ve been working on a tool to scrape urls. The idea is to have a tool that checks my sites for invalid code, urls and images.

    Below some tools and ideas that I might consider using

    • Wget can be used to spider a website (list url’s, not download) with the following command: wget -r –spider -o logfile.txt http://www.myurl.com
    • ruby: open-uri & hapricot.
    • php is not a good idea since scripts can only run for a certain time. (php on commandline is ok)
    • put the results in a mysql database so I can link it with other data

    check for:

    • Valid html
    • 404
    • sitespeed
    • image compression (jpegsnoop for photoshop compression, Smush it)
    • readability
    • pagerank
    • internal links (links on my domain to pages on my domain)
    • external links (links on my domain to external domains)
    • received links (who links to me and what alt text is used, page rank)
    • keywords

    Try to :

    curl into Google analytics and import the results in mysql

    [update]should read[/update]

    Related posts:

    1. Prototype: submit multiple select with ajax call

    ]]>     http://www.westworld.be/a-note-to-self/wget-as-a-spider/feed/ 3