Skip to main content
Glite Dictionary
🧎 Moving on hands and knees 🌐 Searching websites
crawling

crawling

4.9
Automatic programs that browse the web to find and list pages for search engines
  • noun
  • /ˈkrɔːlɪŋ/
  • Texting
translation icon : rastreo
  • Effective web crawling helps improve the visibility of websites in search results.

I looked around, I did the usual kind of web crawling, but what I did find is a lot of people that have made and are making gas from wood.

Examples

  • I looked around, I did the usual kind of web crawling, but what I did find is a lot of people that have made and are making gas from wood.

    MIT OpenCourseWare

  • We can't because YouTube has all these bots that that crawl, everything that ever goes up on there.

  • So we've got web crawling your Google.

  • This is like, you know, deciding what URLs crawl, but how you crawl them.

  • Sounds like a robot spider, but it's actually software which crawls around online and searches for new content.

  • Web crawling is essential for search engines to index new websites.

  • During web crawling, the program visits links to gather and store data efficiently.

How Thorough

probing
1 6.5
crawling
  • Texting
4.9
harvesting
  • Specialized
1 4.3
scraping
  • Texting
2 4.5
scanning
  • Specialized
9 3.7

Surface Forms

crawling singular
crawlings plural

Morphology

crawling = crawl (semi-transparent) = crawl + ing

The computing sense is an extension of the idea of 'moving through' content, so the morphological relation is visible but the technical meaning (automated indexation) requires domain knowledge.

Etymology

Crawling in computing borrows the everyday verb crawl, which means 'to move slowly on the ground'. Small programs are said to 'crawl' the web because they move step by step through pages to find and report them, which is why the process is called crawling.