The Internet Archive discovers and captures web pages through many different web crawls.
At any given time several distinct crawls are running, some for months, and some every day or longer.
View the web archive through the Wayback Machine.
The META tag is used to convey meta-information about the document, but
can also be used to specify headers for the document. You can use either
HTTP-EQUIV or NAME to name the meta-information, but CONTENT must be used
in both cases. By using HTTP-EQUIV,
a server should use the name indicated as a header, with the
specified CONTENT as its value. For example,
Provides keywords for search engines such as Infoseek or Alta Vista.
These are added to the keywords found in the document itself. If you
insert a keyword more than seven times here, the whole tag will be
ignored!
<META NAME="description" CONTENT="This is a site">
Search engines which support the above tag will now display the
text you specify here, rather than the first few lines of text from
the actual document when the document shows up in a search result. You
have about 1,000 characters for your description, but not all these
will be used.
This is a so-called "meta refresh", which on certain browsers causes
the document mentioned in the URL to be loaded after n seconds.
This can be used for slide shows or for often-changing information, but
has some drawbacks. In particular, if you use a value of zero seconds,
the user can no longer go "Back" with his back button. He will be
transferred to the specified URL, and when he presses "back" there,
he will go back to the document with the refresh, which immediately
redirects him to the document he tried to get away from.
<META HTTP-EQUIV="expires" CONTENT="Tue, 20 Aug 1996 14:25:27 GMT">
This indicates that the document containing this META tag will
expire at this date. If the document is requested after this date,
the browser should load a new copy from the server, instead of using
the copy in its cache.
Notes:
Not all servers use the information from META tags to generate
headers, although some browsers will treat what they find in here like
it was a header.
Some search engines have put "blocking" measures in place in order
to defeat so-called "spamming", the practice of putting a large number
of keywords or misleading descriptions in META elements.