Metadata-Version: 2.4
Name: piedomains
Version: 0.2.1
Summary: Predict categories based on domain names and their content
Author-email: Rajashekar Chintalapati <rajshekar.ch@gmail.com>, Gaurav Sood <gsood07@gmail.com>
License: MIT License
Project-URL: Homepage, https://github.com/themains/piedomains
Keywords: domain classification,website categorization,machine learning,content analysis,web scraping,computer vision
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: tqdm==4.66.3
Requires-Dist: beautifulsoup4>=4.0.0
Requires-Dist: pandas==1.4.2
Requires-Dist: nltk==3.9
Requires-Dist: tensorflow<2.16.0,>=2.11.1
Requires-Dist: numpy<2.0.0,>=1.21.0
Requires-Dist: scikit-learn==1.5.0
Requires-Dist: joblib==1.2.0
Requires-Dist: selenium==4.8.0
Requires-Dist: webdriver_manager==3.8.5
Requires-Dist: pillow==10.3.0
Provides-Extra: dev
Requires-Dist: check-manifest; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: isort; extra == "dev"
Provides-Extra: test
Requires-Dist: coverage; extra == "test"
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: pytest-mock; extra == "test"
Requires-Dist: flake8; extra == "test"
Dynamic: license-file

===========================================================================================
piedomains: predict the kind of content hosted by a domain based on domain name and content
===========================================================================================

.. image:: https://github.com/themains/piedomains/actions/workflows/python-package.yml/badge.svg
    :target: https://github.com/themains/piedomains/actions/workflows/python-package.yml
.. image:: https://img.shields.io/pypi/v/piedomains.svg
    :target: https://pypi.python.org/pypi/piedomains
.. image:: https://readthedocs.org/projects/piedomains/badge/?version=latest
    :target: http://piedomains.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status
.. image:: https://static.pepy.tech/badge/piedomains
    :target: https://pepy.tech/project/piedomains

The package infers the kind of content hosted by a domain using the domain name or full URL, the textual content, and the screenshot of the homepage.

We use domain category labels from `Shallalist  <https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/ZXTQ7V>`__ and build our own training dataset by scraping and taking screenshots of the homepage. The final dataset used to train the model is posted on the `Harvard Dataverse <https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/ZXTQ7V>`__.  Python notebooks used to build the models can be found `here <https://github.com/themains/piedomains/tree/55cd5ea68ccec58ab2152c5f1d6fb9e6cf5df363/piedomains/notebooks>`__ and the model files can be found `here <https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/YHWCDC>`__

Installation
--------------
We strongly recommend installing `piedomains` inside a Python virtual environment
(see `venv documentation <https://docs.python.org/3/library/venv.html#creating-virtual-environments>`__)

::

    pip install piedomains

General API
-----------
1. **domain.pred_shalla_cat_with_text(input)**

 - What it does:

  - Predicts the kind of content hosted by a domain based on the domain name or full URL and the HTML content. 
  - The function can use locally stored HTML files or fetch fresh HTML files from the specified URLs. 
  - If you specify a local folder, the function will look for HTML files corresponding to the domain name. 
  - The HTML files must be stored as `domainname.html`. 
  - The function returns a pandas dataframe with predicted labels and corresponding probabilities.

 - Inputs:

  - `input`: list of URLs or domain names. Either `input` or `html_path` must be specified.
  - `html_path`: path to the folder where the HTMLs are stored.  Either `input` or `html_path` must be specified. 
  - `latest`: use the latest model. The default is `True.`
  - Note: The function will by default look for a `html` folder on the same level as model files.

 - Output:

  - Returns a pandas dataframe with the predicted labels and probabilities

 - Sample usage:
   ::
     
     from piedomains import domain
     # URLs and domains can be mixed
     inputs = [
         "forbes.com",
         "https://xvideos.com",
         "last.fm",
         "https://facebook.com/news",
         "bellesa.co",
         "https://marketwatch.com/investing"
     ]
     # with URLs/domains
     result = domain.pred_shalla_cat_with_text(inputs)
     # with html path where htmls are stored (offline mode)
     result = domain.pred_shalla_cat_with_text(html_path="path/to/htmls")
     # with URLs/domains and html path, html_path will be used to store htmls
     result = domain.pred_shalla_cat_with_text(inputs, html_path="path/to/htmls")
     print(result)
 - Sample output:
   ::

                 domain  text_label  text_prob  \
     0      xvideos.com        porn   0.918919   
     1  marketwatch.com     finance   0.627119   
     2       forbes.com        news   0.575000   
     3       bellesa.co        porn   0.962932   
     4     facebook.com  recreation   0.200815   
     5          last.fm       music   0.229545   

                                       text_domain_probs  used_domain_text  \
     0  {'adv': 0.001249639527059502, 'aggressive': 9....              True   
     1  {'adv': 0.001249639527059502, 'aggressive': 9....              True   
     2  {'adv': 0.010590500641848523, 'aggressive': 0....              True   
     3  {'adv': 0.00021545223423966907, 'aggressive': ...              True   
     4  {'adv': 0.006381039197812215, 'aggressive': 0....              True   
     5  {'adv': 0.002181818181818182, 'aggressive': 0....              True   

                                           extracted_text  
     0  xvideos furry ass history mature rough redhead...  
     1  marketwatch gold stocks video chrome economy v...  
     2  forbes featured leadership watch money breakin...  
     3  bellesa audio vixen sensual passionate orgy ki...  
     4    facebook watch messenger portal bulletin oculus  
     5  last twitter music reset company back merchand...  

2. **domain.pred_shalla_cat_with_images(input)**

 - What it does:

  - Predicts the kind of content hosted by a domain based on screenshot of the homepage.  
  - The function can use locally stored screenshots files or fetch fresh screenshots of the homepage.  
  - If you specify a local folder, the function will look for jpegs corresponding to the domain. 
  - The screenshots must be stored as `domainname.jpg`. 
  - The function returns a pandas dataframe with label and corresponding probabilities.

 - Inputs:

  - `input`: list of domains. Either `input` or `image_path` must be specified.
  - `image_path`: path to the folder where the screenshots are stored.  Either `input` or `image_path` must be specified. 
  - `latest`: use the latest model. Default is `True.`
  - Note: The function will by default look for a `images`` folder on the same level as model files.

 - Output:

  - Returns panda dataframe with label and probabilities

 - Sample usage:
   ::
     
     from piedomains import domain
     domains = [
         "forbes.com",
         "xvideos.com",
         "last.fm",
         "facebook.com",
         "bellesa.co",
         "marketwatch.com"
     ]
     # with only domains
     result = domain.pred_shalla_cat_with_images(domains)
     # with image path where images are stored (offline mode)
     result = domain.pred_shalla_cat_with_images(image_path="path/to/images")
     # with domains and image path, image_path will be used to store images
     result = domain.pred_shalla_cat_with_images(domains, image_path="path/to/images")
     print(result)
 - Sample output:
   ::

                 domain image_label  image_prob  \
     0       bellesa.co    shopping    0.366663   
     1     facebook.com        porn    0.284601   
     2  marketwatch.com  recreation    0.367953   
     3      xvideos.com        porn    0.916550   
     4       forbes.com  recreation    0.415165   
     5          last.fm    shopping    0.303097   

                                       image_domain_probs  used_domain_screenshot  
     0  {'adv': 0.0009261096129193902, 'aggressive': 3...                    True  
     1  {'adv': 0.030470917001366615, 'aggressive': 0....                    True  
     2  {'adv': 0.006861348636448383, 'aggressive': 0....                    True  
     3  {'adv': 0.0004964823601767421, 'aggressive': 0...                    True  
     4  {'adv': 0.0016061498317867517, 'aggressive': 8...                    True  
     5  {'adv': 0.007956285960972309, 'aggressive': 0....                    True  

3. **domain.pred_shalla_cat(input)**

 - What it does:

  - Predicts the kind of content hosted by a domain based on a screenshot of the homepage.  
  - The function can use locally stored screenshots and HTMLs or fetch fresh data.  
  - If you specify local folders, the function will look for jpegs corresponding to the domain. 
  - The screenshots must be stored as `domainname.jpg`. 
  - The HTML files must be stored as `domainname.html`. 
  - The function returns a pandas dataframe with the predicted labels and corresponding probabilities.

4. **Archive.org Historical Classification (NEW)**

 - **domain.pred_shalla_cat_archive(input, archive_date)**
 - **domain.pred_shalla_cat_with_text_archive(input, archive_date)**  
 - **domain.pred_shalla_cat_with_images_archive(input, archive_date)**

 - What it does:

  - Predicts content categories using historical snapshots from archive.org
  - Fetches content from the closest available snapshot to the specified date
  - Supports the same analysis as regular functions but with historical data
  - Useful for analyzing how website content has changed over time

 - Inputs:

  - `input`: list of URLs or domain names to classify
  - `archive_date`: target date as 'YYYYMMDD' string (e.g., '20200101' for Jan 1, 2020)
  - `html_path`: optional path for storing archived HTML files
  - `image_path`: optional path for storing archived screenshots
  - `use_cache`: whether to reuse existing archived files
  - `latest`: whether to download latest model version

 - Sample usage:
   ::
     
     from piedomains import domain
     
     # Classify domains using content from January 1, 2020
     domains = ["amazon.com", "facebook.com", "cnn.com"]
     result = domain.pred_shalla_cat_archive(domains, "20200101")
     print(result[["domain", "pred_label", "pred_prob", "archive_date"]])
     
     # Text-only classification from archive
     text_result = domain.pred_shalla_cat_with_text_archive(domains, "20200101")
     
     # Compare different time periods
     old_result = domain.pred_shalla_cat_archive(domains, "20100101")  # 2010
     new_result = domain.pred_shalla_cat_archive(domains, "20200101")  # 2020

 - Inputs:

  - `input`: list of domains. Either `input` or `html_path` must be specified.
  - `html_path`: path to the folder where the screenshots are stored.  Either `input`, `image_path`, or `html_path` must be specified. 
  - `image_path`: path to the folder where the screenshots are stored.  Either `input`, `image_path`, or `html_path` must be specified. 
  - `latest`: use the latest model. Default is `True.`
  - Note: The function will by default look for a `html` folder on the same level as model files.
  - Note: The function will by default look for a `images` folder on the same level as model files.

 - Output

  - Returns panda dataframe with label and probabilities

 - Sample usage:
   ::
     
     from piedomains import domain
     domains = [
         "forbes.com",
         "xvideos.com",
         "last.fm",
         "facebook.com",
         "bellesa.co",
         "marketwatch.com"
     ]
     # with only domains
     result = domain.pred_shalla_cat(domains)
     # with html path where htmls are stored (offline mode)
     result = domain.pred_shalla_cat(html_path="path/to/htmls")
     # with image path where images are stored (offline mode)
     result = domain.pred_shalla_cat(image_path="path/to/images")
     print(result)

 - Sample output:
   ::

                   domain  text_label  text_prob  \
     0      xvideos.com        porn   0.918919   
     1  marketwatch.com     finance   0.627119   
     2       forbes.com        news   0.575000   
     3       bellesa.co        porn   0.962932   
     4     facebook.com  recreation   0.200815   
     5          last.fm       music   0.229545   

                                       text_domain_probs  used_domain_text  \
     0  {'adv': 0.001249639527059502, 'aggressive': 9....              True   
     1  {'adv': 0.001249639527059502, 'aggressive': 9....              True   
     2  {'adv': 0.010590500641848523, 'aggressive': 0....              True   
     3  {'adv': 0.00021545223423966907, 'aggressive': ...              True   
     4  {'adv': 0.006381039197812215, 'aggressive': 0....              True   
     5  {'adv': 0.002181818181818182, 'aggressive': 0....              True   

                                           extracted_text image_label  image_prob  \
     0  xvideos furry ass history mature rough redhead...        porn    0.916550   
     1  marketwatch gold stocks video chrome economy v...  recreation    0.370665   
     2  forbes featured leadership watch money breakin...  recreation    0.422517   
     3  bellesa audio vixen sensual passionate orgy ki...        porn    0.409875   
     4    facebook watch messenger portal bulletin oculus        porn    0.284601   
     5  last twitter music reset company back merchand...    shopping    0.420788   

                                       image_domain_probs  used_domain_screenshot  \
     0  {'adv': 0.0004964823601767421, 'aggressive': 0...                    True   
     1  {'adv': 0.007065971381962299, 'aggressive': 0....                    True   
     2  {'adv': 0.0016623957781121135, 'aggressive': 7...                    True   
     3  {'adv': 0.0008810096187517047, 'aggressive': 0...                    True   
     4  {'adv': 0.030470917001366615, 'aggressive': 0....                    True   
     5  {'adv': 0.01235155574977398, 'aggressive': 0.0...                    True   

           label  label_prob                              combined_domain_probs  
     0      porn    0.917735  {'adv': 0.0008730609436181221, 'aggressive': 0...  
     1   finance    0.315346  {'adv': 0.004157805454510901, 'aggressive': 0....  
     2      news    0.367533  {'adv': 0.006126448209980318, 'aggressive': 0....  
     3      porn    0.686404  {'adv': 0.0005482309264956868, 'aggressive': 0...  
     4      porn    0.223327  {'adv': 0.018425978099589416, 'aggressive': 0....  
     5  shopping    0.232422  {'adv': 0.007266686965796081, 'aggressive': 0....  


Authors
-------
Rajashekar Chintalapati and Gaurav Sood

Contributor Code of Conduct
---------------------------------
The project welcomes contributions from everyone! In fact, it depends on
it. To maintain this welcoming atmosphere, and to collaborate in a fun
and productive way, we expect contributors to the project to abide by
the `Contributor Code of Conduct <http://contributor-covenant.org/version/1/0/0/>`__.

License
----------
The package is released under the `MIT License <https://opensource.org/licenses/MIT>`__.
