Add sponsored search results and Pandas DataFrame conversion #69
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces the ability to fetch sponsored search results from Google. - Extended the `SearchResult` class to include a `is_sponsored` boolean field. - Modified the `search` function to accept a new parameter `sponsored` that toggles whether or not to include sponsored results. - Added additional parsing logic to identify and include sponsored results in the output. Note: Ensure compliance with Google's terms of service when using this feature.
This commit adds a utility function `to_df` that converts the search results to a Pandas DataFrame for easier manipulation and analysis. - Created `to_df` function that takes an iterable of `SearchResult` objects. - The function enumerates through the search results, extracting `url`, `title`, and `description`, and optionally `is_sponsored` if present. - Returns a Pandas DataFrame containing these details. This feature enhances data manipulation capabilities, making it easier to process and analyze search results.
This commit updates the README to reflect the recent changes and new features added to the library: - Added section on fetching sponsored results using the `sponsored` parameter. - Included information about the `to_df` function for converting search results to a Pandas DataFrame. The update aims to provide users with a comprehensive guide to using the latest version of the library.
Nv7-GitHub
requested changes
Nov 9, 2023
| @@ -4,7 +4,7 @@ | |||
| from requests import get | |||
| from .user_agents import get_useragent | |||
| import urllib | |||
|
|
|||
| import pandas as pd | |||
There was a problem hiding this comment.
Did you add this to requirements.txt?
| @@ -52,29 +70,52 @@ def search(term, num_results=10, lang="en", proxy=None, advanced=False, sleep_in | |||
| start = 0 | |||
| while start < num_results: | |||
| # Send request | |||
| resp = _req(escaped_term, num_results - start, | |||
| resp = _req(escaped_term,num_results - start, | |||
|
Nv7-GitHub
reviewed
Nov 22, 2023
| @@ -1,2 +1,3 @@ | |||
| beautifulsoup4>=4.9 | |||
| requests>=2.20 | |||
| pandas | |||
There was a problem hiding this comment.
Probably add a version just to make sure it works in the future and we have less version issues
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds two new features to the library, as suggested in Issue #11:
searchfunction now accepts asponsoredboolean flag to fetch sponsored search results.to_dffunction to convert search results into a Pandas DataFrame.Changes
SearchResultclass to include anis_sponsoredattribute.searchfunction to accommodate thesponsoredflag.to_dffunction to convert search results to a Pandas DataFrame.How to Test
sponsored=Trueflag.to_dffunction.Issue Link
Closes #11