atextcrawler/doc/source/index.rst

38 lines
967 B
ReStructuredText
Raw Normal View History

2021-11-29 09:16:31 +00:00
atextcrawler
============
atextcrawler is an asynchronous webcrawler indexing text
for literal and semantic search.
Its client-side counterpart is atextsearch_.
atextcrawler crawls and indexes selected websites.
It starts from a few seed sites and follows their external links.
Criteria defined in plugin code determine which linked sites (and
which of their resources) are (recursively) added to the pool.
atextcrawler is written in Python, runs a configurable number of
async workers concurrently (in one process), uses tensorflow for
embedding (paragraph-sized) text chunks in a (multi-)language model
and stores metadata in PostgreSQL and texts in elasticsearch.
.. _atextsearch: https://gitea.multiname.org/a-text/atextsearch
.. toctree::
:maxdepth: 2
:caption: Contents:
introduction
installation
maintenance
development
reference/modules
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`