Notes on current search problems, Elasticsearch indexing and re-indexing constraints, and some ONTIC-specific field behavior.

Problems

• Matching: double quotes "" matches are not working for the aboutMe field
• Relevancy: exact matches are sometimes scored lower than partial matches

ES Indexing and Re-indexing

ES Indexing and Re-Indexing

• Not all settings can be changed after index creation
  • You can add new fields and update the index by adding new fields to the mapping
  • You can update dynamic settings like replicas and refresh interval
  • You can add or update index aliases
• But you cannot:
  • Update the mapping of existing fields. For that, you need reindexing
  • Update all settings, for example number_of_shards

Re-indexing Strategy

• Create a new index with updated mappings and settings
• Copy data from the old index to the new index
  • The reindex API reads documents from your source, which is the old index
  • It then indexes these documents into your destination, which is the new index
  • The documents are processed according to the new mappings and settings
• Switch alias or point alias to the new index
  • You can delete the old index if you want

alias approach is essentially a way to redirect requests from a fixed name, the alias, to a specific index

Example:

• myindex_v1
• myindex_v2
• myindex -> alias

Important Re-indexing Caveat

• Reindexing is done in the background and it takes time, proportional to how large the dataset is
• While reindexing is running, if you modify data in the old index, remember:
  • The reindex operation works like a snapshot
  • It only captures the data that existed in the source index at the time the reindex operation started
  • Any documents added, modified, or deleted in the source index after the reindex begins will not be reflected in the destination index

If you need to capture ongoing changes during a migration, you have a few options:

• Pause writes
  • Temporarily redirect writes to a queue or buffer while completing the reindex
• Dual writes
  • Write to both old and new indices during the transition
• Reindex with a time filter
  • Do an initial reindex, then run subsequent reindexes with time filters to capture only newer documents

Ontic

• Process all the documents that are created or modified during the reindexing process
• Reindexing uses the scroll API, which takes a snapshot before reindexing starts, so newly added or modified documents will not be part of that first reindex
• Do multiple syncs based on last modified time to ensure no document gets missed

Entity

• fullName.words - 10 (text / word_token_analyzer)
• relationship.firstname.words - 15
• vin.partial - 50 (text / partial_edge_ngram_token_analyzer)
• aboutMe - 2
• fullName - 10
• corporateOfficerDetails.lastname.partial - 25
• screename.partial - 30
• searchDetails.custom... - 50
• phoneDetails.fullNumber - 50 (keyword)
• weaponType - 20
• email - 30
• ssnDetails - 50
• duns - 50
• email.normalize - 100

Query

Double quotes ""

• aboutMe.words - word_token_analyzer (2)
• aboutMe should likely use a match_phrase query instead of a term query

Questions