Want To Access SQL Database To Run Loop To Check Robots.txt Remotely

One of my tables in the db contains the websites and the location of their link directory. i.e. domain.com/linkdirectory

I want to run a check against every website's robots.txt file to ensure that it doesn't look like this:

user-agent: *

Disallow: /linkdirectory/

so I need a program that does the following:

 For each website in database (start of loop)

  Does robots.txt exist on their site ?(run a remote check)

     if it does

        is their link directory blocked in robots.txt?

          if so, then log their user id and website name into an array for later use

          end if

       endif

end loop

How would I do this in ASP.net 2.0?