Metadata-Version: 2.1
Name: scrapydwebx
Version: 1.5.6
Summary: Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization.
Home-page: https://github.com/payala/scrapydwebx
Author: payala
Author-email: ppayala@gmail.com
License: GNU General Public License v3.0
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: APScheduler==3.6.0
Requires-Dist: click==7.1.2
Requires-Dist: colorama==0.4.0
Requires-Dist: Flask==2.0.0
Requires-Dist: Flask-Compress==1.4.0
Requires-Dist: Flask-SQLAlchemy==2.4.0
Requires-Dist: idna==2.7
Requires-Dist: itsdangerous==2.0.0
Requires-Dist: Jinja2==3.0.0
Requires-Dist: logparser==0.8.2
Requires-Dist: MarkupSafe==2.0.0
Requires-Dist: pexpect==4.7.0
Requires-Dist: ptyprocess==0.6.0
Requires-Dist: pytz==2018.9
Requires-Dist: requests>=2.21.0
Requires-Dist: setuptools>=40.6.3
Requires-Dist: six==1.16.0
Requires-Dist: SQLAlchemy==1.3.24
Requires-Dist: tzlocal==1.5.1
Requires-Dist: w3lib==2.0.0
Requires-Dist: Werkzeug==2.0.0

English | [简体中文](https://github.com/my8100/scrapydweb/blob/master/README_CN.md)

# ScrapydWeb: Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization.

[![PyPI - scrapydweb Version](https://img.shields.io/pypi/v/scrapydweb.svg)](https://pypi.org/project/scrapydweb/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/scrapydweb.svg)](https://pypi.org/project/scrapydweb/)
[![CircleCI](https://circleci.com/gh/my8100/scrapydweb/tree/master.svg?style=shield)](https://circleci.com/gh/my8100/scrapydweb/tree/master)
[![codecov](https://codecov.io/gh/my8100/scrapydweb/branch/master/graph/badge.svg)](https://codecov.io/gh/my8100/scrapydweb)
[![Coverage Status](https://coveralls.io/repos/github/my8100/scrapydweb/badge.svg?branch=master)](https://coveralls.io/github/my8100/scrapydweb?branch=master)
[![Downloads - total](https://static.pepy.tech/badge/scrapydweb)](https://pepy.tech/project/scrapydweb)
[![GitHub license](https://img.shields.io/github/license/my8100/scrapydweb.svg)](https://github.com/my8100/scrapydweb/blob/master/LICENSE)
[![Twitter](https://img.shields.io/twitter/url/https/github.com/my8100/scrapydweb.svg?style=social)](https://twitter.com/intent/tweet?text=@my8100_%20ScrapydWeb:%20Web%20app%20for%20Scrapyd%20cluster%20management,%20with%20support%20for%20Scrapy%20log%20analysis%20%26%20visualization.%20%23python%20%23scrapy%20%23scrapyd%20%23webscraping%20%23scrapydweb%20&url=https%3A%2F%2Fgithub.com%2Fmy8100%2Fscrapydweb)


##
![servers](https://raw.githubusercontent.com/my8100/scrapydweb/master/screenshots/servers.png)

## Scrapyd ScrapydWeb LogParser
### Recommended Reading
[How to efficiently manage your distributed web scraping projects](https://github.com/my8100/files/blob/master/scrapydweb/README.md)

[How to set up Scrapyd cluster on Heroku](https://github.com/my8100/scrapyd-cluster-on-heroku)


## Demo
[scrapydweb.herokuapp.com](https://scrapydweb.herokuapp.com)


## Features
<details>
<summary>View contents</summary>

- Scrapyd Cluster Management
  - All Scrapyd JSON API Supported
  - Group, filter and select any number of nodes
  - **Execute command on multinodes with just a few clicks**

- Scrapy Log Analysis
  - Stats collection
  - **Progress visualization**
  - Logs categorization

- Enhancements
  - **Auto packaging**
  - **Integrated with [*LogParser*](https://github.com/my8100/logparser)**
  - **Timer tasks**
  - :e-mail: **Monitor & Alert**
  - Mobile UI
  - Basic auth for web UI

</details>


## Getting Started
<details>
<summary>View contents</summary>

### Prerequisites
**Make sure that [Scrapyd](https://github.com/scrapy/scrapyd) has been installed and started on all of your hosts.**

Note that for remote access, you have to manually set 'bind_address = 0.0.0.0' in [the configuration file of Scrapyd](https://scrapyd.readthedocs.io/en/latest/config.html#example-configuration-file)
and restart Scrapyd to make it visible externally.

### Install
- Use pip:
```bash
pip install scrapydweb
```
Note that you may need to execute `python -m pip install --upgrade pip` first in order to get the latest version of scrapydweb, or download the tar.gz file from https://pypi.org/project/scrapydweb/#files and get it installed via `pip install scrapydweb-x.x.x.tar.gz`

- Use git:
```bash
pip install --upgrade git+https://github.com/my8100/scrapydweb.git
```
Or:
```bash
git clone https://github.com/my8100/scrapydweb.git
cd scrapydweb
python setup.py install
```

### Start
1. Start ScrapydWeb via command `scrapydweb`. (a config file would be generated for customizing settings at the first startup.)
2. Visit http://127.0.0.1:5000 **(It's recommended to use Google Chrome for a better experience.)**

### Browser Support
The latest version of Google Chrome, Firefox, and Safari.

</details>


## Running the tests
<details>
<summary>View contents</summary>

<br>

```bash
$ git clone https://github.com/my8100/scrapydweb.git
$ cd scrapydweb

# To create isolated Python environments
$ pip install virtualenv
$ virtualenv venv/scrapydweb
# Or specify your Python interpreter: $ virtualenv -p /usr/local/bin/python3.7 venv/scrapydweb
$ source venv/scrapydweb/bin/activate

# Install dependent libraries
(scrapydweb) $ python setup.py install
(scrapydweb) $ pip install pytest
(scrapydweb) $ pip install coverage

# Make sure Scrapyd has been installed and started, then update the custom_settings item in tests/conftest.py
(scrapydweb) $ vi tests/conftest.py
(scrapydweb) $ curl http://127.0.0.1:6800

# '-x': stop on first failure
(scrapydweb) $ coverage run --source=scrapydweb -m pytest tests/test_a_factory.py -s -vv -x
(scrapydweb) $ coverage run --source=scrapydweb -m pytest tests -s -vv --disable-warnings
(scrapydweb) $ coverage report
# To create an HTML report, check out htmlcov/index.html
(scrapydweb) $ coverage html
```

</details>


## Built With
<details>
<summary>View contents</summary>

<br>

- Front End
  - [Element](https://github.com/ElemeFE/element)
  - [ECharts](https://github.com/apache/incubator-echarts)

- Back End
  - [Flask](https://github.com/pallets/flask)

</details>


## Changelog
Detailed changes for each release are documented in the [HISTORY.md](https://github.com/my8100/scrapydweb/blob/master/HISTORY.md).


## Author
| [<img src="https://github.com/my8100.png" width="100px;"/>](https://github.com/my8100)<br/> [<sub>my8100</sub>](https://github.com/my8100) |
| --- |


## Contributors
| [<img src="https://github.com/simplety.png" width="100px;"/>](https://github.com/simplety)<br/> [<sub>Kaisla</sub>](https://github.com/simplety) |
| --- |


## License
This project is licensed under the GNU General Public License v3.0 - see the [LICENSE](https://github.com/my8100/scrapydweb/blob/master/LICENSE) file for details.
