Tag Archives: crawler

Professionаl program for creating, editing and submitting XML sitemaps. Sitemap Writer Pro is a professional xml sitemap creator program. Automatically generate sitemaps for your websites in xml or compressed gzip format, keep up-to-date, edit them, upload to web server and submit to search engines that support sitemaps.

Downloads one or thousands of sites into packed files (saving up to 80% of disk WebCloner is a site ripper, images and Flash downloader, offline browser, and e-book compiler. You can use it to scan, download, browse and convert to e-books (Self running executables or CHM) entire websites or just parts of them.

Multithreaded Linux-based crawling system including numerous custom crawlers.

Crawls, searches the internet for file types that you specify Crawls, searches the internet for file types that you specify. Tell it what site or sites to start with and what type of files you are looking for and let it go