Every time you post a new article or upload a new video, you are creating a new object on the web. Google will index its URL, and Facebook will add it to the Open Graph.
Keeping track of your web objects inventory can help you to size how big is your website (the more content, the better) and check how it is indexed in Google, Facebook or any other social network.
Here are the installation steps:
pip3 install scrapy
scrapy startproject links_mapper
scrapy genspider leocelis leocelis.com
Once you are done you can run:
scrapy crawl leocelis -o links.csv -t csv
This last command will generate a links.csv file with a list of all the URLs and titles from the domain you’ve specified.