Web scraper / parser and crawler

hello,

what are the bests Web scrapers/parsers and crawlers?

Thank you

 

bump!

 

what exactly do u need 777

 

parse links from search engine and spider http websites..

 

dig into github.com. 4 instance, https://github.com/celrenheit/spider

 

can you suggest other? for CLI
maybe you can ask someone or search something on some forum?

Thanks!

 

phakt, hmmm.. not sure what's exactly u need. actually, it's quite possible to use lynx+grep to build any spider/scraper.

 

can you give any pratical examples?
any software or bash script that does this?

 

phakt, https://www.gnu.org/software/wget/manual/html_node/Recursive-Download.html https://www.linuxjournal.com/content/downloading-entire-web-site-wget

 

where can i see sources ready for compile & test of spider and scraper for example in Python?

Thanks