search
******

.. highlight:: python

https://gitlab.com/kb/search

See the :doc:`dev-elasticsearch` documentation for configuration and Salt
installation.

Index
=====

The ``search`` app has a ``SearchIndex`` class in ``search/search.py``.
This class does the hard work of searching.

As a developer, you just need to create a new class for each of your indexes.
Sample index classes are here:

1. ``ContactSearch`` in
   https://gitlab.com/kb/contact/blob/master/contact/search.py
2. ``TicketSearch`` in
   https://gitlab.com/kb/crm/blob/master/crm/search.py

ElasticSearch 8.x
-----------------

Changes required for this version include e.g. ``ContactSearch``

1. Remove the ``document_type`` property (look in the ``Mixin`` class).
2. Rename ``_settings`` to ``settings``
3. Rename ``configuration`` to ``mappings`` and make it a method
   (not a ``property``)
4. ``mappings`` to return just the ``properties``
   i.e. remove the ``settings``, ``mappings`` and ``document_type`` keys.

For sample code, see:

- https://gitlab.com/kb/crm/-/commit/16cf85bbc7d6707f07fd53e3f8203cb15f62a4fe
- https://gitlab.com/kb/contact/-/commit/da645786e371ff6a88a67d41af24f4b95a131658
- https://gitlab.com/kb/contact/-/commit/d6ad4c40596cceeb44ea4b535a857d0510f7684a

Delete
------

We have updated our strategy to include deleted rows in the index, with the
option to include them in a search.

For more information, on our delete strategy, see the following links:

1. https://www.kbsoftware.co.uk/crm/ticket/2790/
2. https://gitlab.com/kb/contact/commit/b11f6842bccf0e04cbc9425c5425363f2542b842

View
====

The ``search`` app also has ``SearchViewMixin`` which can be used to search the
indexes e.g:: ``dash.SearchView`` in
https://gitlab.com/kb/kbsoftware_couk/blob/master/dash/views.py#L66
which uses ``ContactSearch`` and ``TicketSearch`` (defined above).

Rebuild and Refresh
===================

To rebuild your index, create a task e.g. ``rebuild_contact_index`` in
https://gitlab.com/kb/contact/blob/master/contact/tasks.py

Create a management command to call the ``rebuild_contact_index`` task:
e.g. `rebuild_contact_index.py`_

To **refresh** your index, use the same idea, but call ``refresh`` instead e.g::

  index = SearchIndex(ContactIndex())
  count = index.refresh()

Search
======

Issues are much easier to diagnose if you can run a simple management command to
perform a search query:

Create a management command called ``search_contact_index``
e.g. `search_contact_index.py`_

Update
======

Create an index update function e.g:: ``update_contact_index`` in
https://gitlab.com/kb/contact/blob/master/contact/tasks.py

In your create and update views, call the update task e.g::

  from django.db import transaction
  from contact.tasks import update_contact_index

  transaction.on_commit(lambda: update_contact_index.delay(self.object.pk))

For a simple ``UpdateView``, the minimum viable code is as follows::

  def form_valid(self, form):
      result = super().form_valid(form)
      transaction.on_commit(lambda: update_contact_index.delay(self.object.pk))
      return result

ElasticSearch
=============

Prerequisites
-------------

Setup :doc:`celery`...

Install
-------

Follow the `ElasticSearch - Getting Started`_ instructions...

In your ``requirements/base.txt``, add the following::

  elasticsearch

.. tip:: Find the version number in :doc:`dev-requirements`

In ``settings/production.py`` (after ``CELERY_DEFAULT_QUEUE``)::

  from celery.schedules import crontab
  CELERYBEAT_SCHEDULE = {
      'rebuild_contact_index': {
          'task': 'contact.tasks.rebuild_contact_index',
          'schedule': crontab(minute='10', hour='1'),
      },
  }

.. note:: Remember to use the correct pattern for transactions when queuing
          search index updates.  For details, see :ref:`django_transactions`

Diagnostics
===========

Analyze
-------

To understand how your field is being analyzed (this example is from the
:doc:`app-contact` app)::

  from search.search import SearchIndex
  from contact.search import ContactIndex

  index = SearchIndex(ContactIndex())
  index.drop_create()

  import json
  print(json.dumps(index.analyze('autocomplete', 'EX2 2AB'), indent=4))
  print(json.dumps(index.analyze('autocomplete_search', 'EX2 2AB'), indent=4))

.. note:: This example uses the ``ContactSearch`` index and the ``autocomplete``
          analyzers:
          https://gitlab.com/kb/contact/blob/master/contact/search.py#L61

For other diagnostics, see :doc:`diagnostics`...

Explain
-------

To understand the score for search results:

1. Make sure ``DEBUG`` is set to ``True`` in your ``settings``.
2. Add ``explain`` to your call to the ``search`` method.

e.g::

  result = search_index.search(
      criteria,
      explain=True,
  )

.. tip:: You could add this to the ``SearchViewMixin`` class
         in ``search/views.py``.

The ``_explain`` method in ``search/search.py`` will write a time-stamped file
containing the results e.g. ``elastic-explain-2019-01-07-13-20-46.json``.

Maintenance
===========

To manually update the index run the management command created earlier
(see `rebuild_contact_index.py`_).

The flush process of an index basically frees memory::

  curl localhost:9200/_flush

Test
====

To check the install::

  curl -X GET 'http://localhost:9200/?pretty'

Query
=====

.. note:: Replace ``hatherleigh_info`` with your site name.

Install ``httpie``::

  pip install httpie

Create a ``json`` file containing your query e.g. ``query.json``::

  {
      "query": {
          "match": {
              "part": "B020"
          }
      }
  }

In this example, we are searching for ``B020``.

Run the query::

  http GET http://localhost:9200/hatherleigh_info/_search < query.json

Explain
-------

To explain the query above::

  http GET "http://localhost:9200/hatherleigh_info/part/_validate/query?explain" < query.json

The ``part`` is the document type (``DOC_TYPE`` in the index ``mappings``
below):

.. code-block:: python

  es.indices.create(
       SEARCH_INDEX,
       {
           'mappings': {
               self.DOC_TYPE: {
                   "properties": {
                       "part": {
                           "type": "string",
                           "analyzer": "autocomplete",
                       },

Analyze
-------

See Diagnostics_ above... or:

Create a ``json`` file containing your query e.g. ``analyze.json``::

  {
      "analyzer": "autocomplete",
      "text": "quick brown"
  }

Run the analysis::

  http GET http://localhost:9200/hatherleigh_info/_analyze < analyze.json


.. _`ElasticSearch - Getting Started`: https://www.pkimber.net/howto/java/apps/elasticsearch/getting-started.html
.. _`rebuild_contact_index.py`: https://gitlab.com/kb/contact/blob/master/contact/management/commands/rebuild_contact_index.py
.. _`search_contact_index.py`: https://gitlab.com/kb/contact/blob/master/contact/management/commands/search_contact_index.py