useful tools when performing collection of information

anew

Append lines from stdin to a file, but only if they don't already appear in the file. Outputs new lines to stdouttoo, making it a bit like a tee -a that removes duplicates.

Link: https://github.com/tomnomnom/anew

Hakrawler

Fast golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library.

Link: https://github.com/hakluke/hakrawler