Loading practice workspace...

For the best experience, increase the window size or view on a laptop or desktop device

Last updated: Pending

Title
Loading...

1/1

No questions are available yet

Web Crawler Design | Databricks Interview Question | InterviewDB

Back to all Databricks interview questions

Desktop gives you the best reading and discussion experience. On smaller screens the description and discussion stack automatically.

Last reported 3 years ago

Web Crawler Design

System DesignOnsite

Objective: Develop a program to perform web crawling starting from a given URL (start_url) up to a specified depth (K). The program should download web pages and store them in a directory structure.

Core Features

Web Crawling:
- Start from start_url and recursively visit links found on the page.
- Limit the crawling to K levels deep from the start_url.
Directory Management:
- Create a directory structure to store the downloaded content.
- Each webpage can be stored as a file or in a sub-directory, based on its URL and depth.
Loop and Duplicate Handling:
- Keep track of visited URLs to avoid revisiting the same page (deduplication).
- Implement a mechanism to detect and handle loops.

Follow-Up

How to handle long fetch time and what about doing it in a distributed way?

Discuss

Community discussion, answers, and follow-up details for this question.

Sort by

Loading comments...

Last reported 3 years ago

Web Crawler Design

System DesignOnsite

Core Features

Web Crawling:
- Start from start_url and recursively visit links found on the page.
- Limit the crawling to K levels deep from the start_url.
Directory Management:
- Create a directory structure to store the downloaded content.
- Each webpage can be stored as a file or in a sub-directory, based on its URL and depth.
Loop and Duplicate Handling:
- Keep track of visited URLs to avoid revisiting the same page (deduplication).
- Implement a mechanism to detect and handle loops.

Follow-Up

How to handle long fetch time and what about doing it in a distributed way?

Discuss

Community discussion, answers, and follow-up details for this question.

Sort by

Loading comments...

Interview Resources

Interview Questions By Company Interview Guides 1point3acres Content Request Leetcode Tagged Questions Request

InterviewDB

Features Testimonials Terms of Service

About InterviewDB

Crowdsourced Interview Question Bank for Job Seekers in Tech/Finance Industry

Discord Community Email Us