Content discovery refers to anything of possible value. Such as pictures, videos, files, backup files, features.
When it comes to content discovery there are 3 methods to doing so.
When it comes to webpages there is always a txt file that tells search engines what to show and what not to show to the public and what we are referring to is the robots.txt file.
So, let’s take a look at the robots.txt for this challenge and see what we find.
As you can see when we access the robots.txt file we see what the public is allowed to see which is the root directory / and what the public isn’t allowed to see /staff-portal.
Now, we know just because it says it is disallowed it doesn’t mean it isn’t accessible it just isn’t shown to the public.
For this challenge you’ll need open up your terminal if you are using a Linux box and enter in curl https://static-labs.tryhackme.cloud/sites/favicon/images/favicon.ico | md5sum.
