A system installed on a server which will allow me to :

A. Upload a list of a few thousand website URLS (csv or text).
I’d like to update the list from time to time by uploading a new file or, if it doesn’t make it complicated, manage a few lists.

B. Manage a list of strings (or key/values) to look for on the website list.
For example, a string can be "WooCommerce" and the tag for it will be "Woocommerce ecommerce system"

C. Every 2 days the system will run on the entire list and perform the following actions:

C.1. Per each domain on the list the system will make a few page requests with a few payloads that I’ll provide (query strings and http post). It will then parse the http response to find one of strings on B and if found, will tag the URL accordingly

C.2. It will make a DNS query to each domain on the list and will write the IP address it resolves to.  If possible, also the whois info (organization name owning the IP)

D. On a designated URL I’ll be able to see a list of all the domains or download a CSV/XSLs.
The structure will be:
URL | IP address | Org name | Tags

Budget: $2,000

Posted On: January 21, 2024 06:20 UTC
Category: Scripting & Automation
Skills:HTTPS, DNS, Web Crawling

Country: Israel

click to apply

Powered by WPeMatico