KB Software Documentation
0.1.01

Standards

  • Glossary
  • Standards
  • Django / Python Code Standards
  • Documentation
  • Ember / JavaScript Code Standards
  • Security Standards - Development
  • Sys-Admin and Configuration Management
  • Technology Standards
  • UI Standards

Sales

  • Sales

Process

  • Process
  • Development Environment
  • Checklist
  • MatterMost
  • Monitor - Elastic and Raygun
  • Workflow

Customer Support

  • SEO
  • Android
  • Backup (PC/Laptop Repair)
  • Google Apps
  • postfix
  • Security - Sys-Admin
  • VPN

Master

  • Salt Master
  • devpi

Minion

  • Salt Cloud - Install
  • Salt - Provision
  • Fabric
  • Database
  • Deploy
  • Deploy on Windows
  • Release
  • Search
  • Install SSL Certificate

System Administration

  • Diagnostics
  • Issues
  • Microsoft Azure
  • Keycloak
  • MatterMost - Install and Update
  • Microsoft (Group Policies)
  • NoMachine
  • Microsoft Office 365
  • Postgres
  • Ubuntu

Site

  • Alfresco Community Edition
  • Backup
  • Backup Dropbox
  • Cache
  • Site - Configuration
  • Pillar
  • Rename a Digital Ocean droplet
  • Site Maintenance Mode
  • Authentication
  • Celery (using Redis)
  • Firewall
  • FTP
  • Google
  • Mailgun
  • Maintenance
  • Restore (as a task)
  • SOLR
  • Create SSL Certificate
  • Obtain and configure ssl certificate on nginx with letsencrypt
  • Remove a site from a server

Development - Django

  • Django
  • Django Applications
  • Django Auth
  • Django Field Types
  • Django Forms
  • GeoDjango
  • Django Media
  • Django Migrations
  • Django Reversion
  • Django Static
  • Django Testing
  • Django Thumbnails
  • Django View
  • Django on Windows

Development

  • Journal
  • Learning
  • APScheduler (with Redis)
  • Development Environment
  • Captcha
  • Chosen - Select Box
  • CKEditor
  • CodeMirror Editor
  • Cookie Confirm
  • Debug
  • Django Debug Toolbar
  • Dramatiq (using Redis)
  • ElasticSearch
  • Ember
  • Ember AddOns
  • Ember - KB Addons
  • Ember Fields
  • Ember Patterns
  • Deploy a project for the first time
  • Fish Shell
  • Flowable
  • git
  • GitLab
  • grunt
  • Google Development
  • LESS
  • Logging
  • Magento XMLRPC API
  • Move a site from Sqlite to Postgresql
  • Pagination
  • PDF Object
  • pytest
  • Requirements
  • Django Rest Framework
  • Restore
  • S3
  • sass
  • Scripts to aid the development process
  • Shiny
  • SiteMap
  • Social
  • Tailwind
  • Testing
  • Time zones
  • Development Tools
  • Web Fonts and favicons
  • Domain

Apps

  • api
  • Apps
  • base
  • block
  • Blog
  • booking
  • Chat
  • Checkout (with Stripe SCA)
  • compose
  • Contact
  • crm
  • enquiry
  • Finance
  • Image Gallery
  • GDPR
  • invoice
  • Job
  • login
  • Magento App
  • mail
  • MailChimp
  • Monitor
  • Microsoft Graph
  • Pipeline
  • Q and A
  • Record Management
  • Report
  • Sales Order
  • search
    • Index
      • Delete
    • View
    • Rebuild and Refresh
    • Search
    • Update
    • ElasticSearch
      • Prerequisites
      • Install
    • Diagnostics
      • Analyze
      • Explain
    • Maintenance
    • Test
    • Query
      • Explain
      • Analyze
  • WooCommerce
  • Work
  • Workflow
  • Xero App

Cloud

  • Digital Ocean

Kubernetes

  • Development with Kubernetes
  • Kubernetes for Development - Install

Project

  • Simple CMS

PHP

  • PHP
  • Drupal

Detailed Notes

  • devpi (in detail)
  • Restore (in detail)
  • LetsEncrypt - DNS
  • Obtain and configure ssl certificate on nginx with letsencrypt - Manual Configuration
  • VPN

Old Notes

  • Checkout (before Stripe SCA)
  • pay
  • Old Activiti Notes
  • Adding Angular to Django Project
  • Backup
  • Development with Docker
  • Kubernetes (for development)
  • Old Notes
  • Database
  • Files
  • Backup
  • Monitor
  • Restore - Server
  • Amazon
  • Amazon Database
  • Amazon S3
  • Digital Ocean
  • Salt Cloud - Install (Legacy Notes)
  • Rackspace
  • Salt Masterless
KB Software Documentation
  • »
  • search
  • View page source

search¶

https://gitlab.com/kb/search

See the doc:dev-elasticsearch documentation for configuration and Salt installation.

Index¶

The search app has a SearchIndex class in search/search.py. This class does the hard work of searching.

As a developer, you just need to create a new class for each of your indexes. Sample index classes are here:

  1. ContactSearch in https://gitlab.com/kb/contact/blob/master/contact/search.py

  2. TicketSearch in https://gitlab.com/kb/crm/blob/master/crm/search.py

Delete¶

We have updated our strategy to include deleted rows in the index, with the option to include them in a search.

For more information, on our delete strategy, see the following links:

  1. https://www.kbsoftware.co.uk/crm/ticket/2790/

  2. https://gitlab.com/kb/contact/commit/b11f6842bccf0e04cbc9425c5425363f2542b842

View¶

The search app also has SearchViewMixin which can be used to search the indexes e.g:: dash.SearchView in https://gitlab.com/kb/kbsoftware_couk/blob/master/dash/views.py#L66 which uses ContactSearch and TicketSearch (defined above).

Rebuild and Refresh¶

To rebuild your index, create a task e.g. rebuild_contact_index in https://gitlab.com/kb/contact/blob/master/contact/tasks.py

Create a management command to call the rebuild_contact_index task: e.g. rebuild_contact_index.py

To refresh your index, use the same idea, but call refresh instead e.g:

index = SearchIndex(ContactIndex())
count = index.refresh()

Search¶

Issues are much easier to diagnose if you can run a simple management command to perform a search query:

Create a management command called search_contact_index e.g. search_contact_index.py

Update¶

Create an index update function e.g:: update_contact_index in https://gitlab.com/kb/contact/blob/master/contact/tasks.py

In your create and update views, call the update task e.g:

from django.db import transaction
from contact.tasks import update_contact_index

transaction.on_commit(lambda: update_contact_index.delay(self.object.pk))

For a simple UpdateView, the minimum viable code is as follows:

def form_valid(self, form):
    result = super().form_valid(form)
    transaction.on_commit(lambda: update_contact_index.delay(self.object.pk))
    return result

ElasticSearch¶

Prerequisites¶

Setup Celery (using Redis)…

Install¶

Follow the ElasticSearch - Getting Started instructions…

In your requirements/base.txt, add the following:

elasticsearch

Tip

Find the version number in Requirements

In settings/production.py (after CELERY_DEFAULT_QUEUE):

from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
    'rebuild_contact_index': {
        'task': 'contact.tasks.rebuild_contact_index',
        'schedule': crontab(minute='10', hour='1'),
    },
}

Note

Remember to use the correct pattern for transactions when queuing search index updates. For details, see Transactions

Diagnostics¶

Analyze¶

To understand how your field is being analyzed (this example is from the Contact app):

from search.search import SearchIndex
from contact.search import ContactIndex

index = SearchIndex(ContactIndex())
index.drop_create()

import json
print(json.dumps(index.analyze('autocomplete', 'EX2 2AB'), indent=4))
print(json.dumps(index.analyze('autocomplete_search', 'EX2 2AB'), indent=4))

Note

This example uses the ContactSearch index and the autocomplete analyzers: https://gitlab.com/kb/contact/blob/master/contact/search.py#L61

For other diagnostics, see Diagnostics…

Explain¶

To understand the score for search results:

  1. Make sure DEBUG is set to True in your settings.

  2. Add explain to your call to the search method.

e.g:

result = search_index.search(
    criteria,
    explain=True,
)

Tip

You could add this to the SearchViewMixin class in search/views.py.

The _explain method in search/search.py will write a time-stamped file containing the results e.g. elastic-explain-2019-01-07-13-20-46.json.

Maintenance¶

To manually update the index run the management command created earlier (see rebuild_contact_index.py).

The flush process of an index basically frees memory:

curl localhost:9200/_flush

Test¶

To check the install:

curl -X GET 'http://localhost:9200/?pretty'

Query¶

Note

Replace hatherleigh_info with your site name.

Install httpie:

pip install httpie

Create a json file containing your query e.g. query.json:

{
    "query": {
        "match": {
            "part": "B020"
        }
    }
}

In this example, we are searching for B020.

Run the query:

http GET http://localhost:9200/hatherleigh_info/_search < query.json

Explain¶

To explain the query above:

http GET "http://localhost:9200/hatherleigh_info/part/_validate/query?explain" < query.json

The part is the document type (DOC_TYPE in the index mappings below):

es.indices.create(
     SEARCH_INDEX,
     {
         'mappings': {
             self.DOC_TYPE: {
                 "properties": {
                     "part": {
                         "type": "string",
                         "analyzer": "autocomplete",
                     },

Analyze¶

See Diagnostics above… or:

Create a json file containing your query e.g. analyze.json:

{
    "analyzer": "autocomplete",
    "text": "quick brown"
}

Run the analysis:

http GET http://localhost:9200/hatherleigh_info/_analyze < analyze.json
Next Previous

© Copyright 2016, KB Software Ltd.

Built with Sphinx using a theme provided by Read the Docs.