Gentics CMS Documentation

Chapters

1. Requirements

2. Installation and Configuration of Elasticsearch

3. Configuration of the CMS

   • Indexing file contents

   • Configuring word decompounder for German

4. Usage

5. Search Index Maintenance

6. REST API

7. Mapping

   • Page

   • Folder

   • Image

   • File

8. New UI

9. Differences between regular and advanced search

1 Requirements

This feature requires an existing installation of Elasticsearch 5 or Elasticsearch 6. Please consult the Elasticsearch documentation for details on installation, configuration and maintenance of Elasticsearch.

For indexing file contents, the Ingest Attachment Plugin is also required.

2 Installation and Configuration of Elasticsearch

The Elasticsearch setup documentation includes information on how to setup Elasticsearch and get it running.

3 Configuration of the CMS

Indexing and searching must be activated and configured in the CMS:

// activate indexing with elasticsearch
$FEATURE["elasticsearch"] = true;

// url for connecting to elasticsearch instance
$ELASTICSEARCH["url"] = "http://localhost:9200/";

// prefix for indices in elasticsearch
$ELASTICSEARCH["index"] = "genticscms";

// number of threads for indexing
$ELASTICSEARCH["threads"] = 10;

3.1 Indexing file contents

For files (but not images), the CMS will also put the content into the Elasticsearch Index to make it searchable. This does not make sense for some files and can lead to resource exhaustion in the Elasticsearch (like Out Of Memory Errors), and can therefore be controlled with black- and whitelists for the mimetypes. It is also possible to further restrict the number of indexed characters (the default limit in Elasticsearch is 100000 characters).

// blacklist 
$ELASTICSEARCH["content"]["blacklist"] = array(
    "application/zip",
    "audio/.*",
    "video/.*");

// whitelist
$ELASTICSEARCH["content"]["whitelist"] = ".*";

// restrict indexed characters
$ELASTICSEARCH["content"]["indexedChars"] = 10000;

• blacklist and whitelist can be single regular expressions or array of regular expressions
• the default blacklist contains application/zip, audio/.* and video/.* (like in the example shown above), the default whitelist is empty (allowing all mimetypes, which are not blacklisted).
• blacklist is stronger than whitelist, if a mimetype matches one of the blacklist patterns, the contents will not be indexed, even if it also matches one of the whitelist patterns.
• indexedChars is per default empty and the default limit in ElasticSearch is 100000 characters.

3.2 Configuring word decompounder for German

For a better search experience, it is advisable to use a word decompounder for analysis of german contents, e.g. the Hyphenation decompounder token filter

The following steps are necessary to accomplish this:

3.2.1 Step 1 – Hyphenations pattern file

The hyphenations pattern file can be downloaded from the OFFO Sourceforge project

From the zip, only the file offo-hyphenation/hyph/de_DR.xml is used and should be placed under analysis/de_DR.xml in the config directory of the Elasticsearch installation.

3.2.2 Step 2 – German dictionary file

The german dictionary file german-dictionary.txt was generated from the igerman98 project.

The wordlist must be placed under analysis/german-dictionary.txt in the config directory of the Elasticsearch installation.

3.2.3 Step 3 – Configuration of index settings

The index settings for the index genticscms_page_de can be changed like this:

// change index setting of pages in german to use a word decompounder
$ELASTICSEARCH["settings"]["page_de"] = array(
    "analysis" => array(
        "analyzer" => array(
            "content_analyzer" => array(
                "type" => "custom",
                "char_filter" => array("html_strip"),
                "tokenizer" => "standard",
                "filter" => array("lowercase", "german_stop", "german_decompounder", "german_snowball")
            ),
            "search_analyzer" => array(
                "type" => "custom",
                "char_filter" => array("html_strip"),
                "tokenizer" => "standard",
                "filter" => array("lowercase", "german_stop", "german_decompounder", "german_snowball")
            )
        ),
        "filter" => array(
            "german_stop" => array(
                "type" => "stop",
                "stopwords" => "_german_"
            ),
            "german_decompounder" => array(
                "type" => "hyphenation_decompounder",
                "word_list_path" => "analysis/german-dictionary.txt",
                "hyphenation_patterns_path" => "analysis/de_DR.xml",
                "min_subword_size" => 4
            ),
            "german_snowball" => array(
                "type" => "snowball",
                "language" => "German"
            )
        )
    )
);

The index setting contains two analyzers, named content_analyzer and the search_analyzer. Both analyzers can be configured identically, or differently depending on the desired indexing and search behaviour.

3.2.4 content_analyzer

The content_analyzer will be used for analysis of the indexed content.

3.2.5 search_analyzer

The search_analyzer will be used for analysis of the search terms.

4 Usage

When the CMS is started (or when the configuration is reloaded), the CMS will check for existence of the required indices and will create missing indices:

Also the pipeline for ingesting attachments will be created, if necessary.

If at least one index is created, an automatic full reindex run is triggered.

5 Search Index Maintenance

The Content.Admin in the old UI contains a new entry Search Index Maintenance which provides an overview over required indices and their status.

The list will indicate indices, which are not fully functional, either because they do not exist, have incorrect settings or mapping, or do not contain the expected number of documents.

The action Rebuild index will update an incorrect mapping, will remove superfluous documents from the index and will reindex every object.

The action Drop and rebuild index will drop, recreate and fill the index.

6 REST API

POST requests to the REST endpoint /CNPortletapp/rest/elastic will be forwarded to the search API endpoint of Elasticsearch for all applicable indices:

E.g. the request

POST http://[cmshost]/CNPortletapp/rest/elastic/_search?sid=[SID]
{
	"_source": false,
	"query": {
		"query_string": {
			"query": "Test*"
		}
	}
}

will be forwarded to

POST http://localhost:9200/genticscms_page,genticscms_page_de,genticscms_page_en,genticscms_file,genticscms_image,genticscms_folder/_search?sid=[SID]
{
	"_source": false,
	"query": {
		"query_string": {
			"query": "Test*"
		}
	}
}

The response will be extended to contain the CMS objects as attribute _object of the returned hits.

7 Mapping

7.1 Page

7.2 Folder

7.3 Image

7.4 File

8 New UI

If the feature is activated, the new UI will automatically use the search index for searching objects. Additionally, there will be an “advanced search” panel below the main search bar, which allows further filtering of results by various parameters.

9 Differences between regular and advanced search

While the regular search just finds contents that contain the exact characters entered by the editor (anywhere in the text, even mid word), search with Elasticsearch works completely different.

When content is indexed in Elasticsearch, it is analyzed with the configured content_analyzer. The analyzer will transform the content into a set of tokens which are put into the index.

While searching, the search term entered by the user is analyzed with the configured search_analyzer (which is by default identical to the content_analyzer). The analyzer will also transform the search term into a set of tokens and Elasticsearch will find matching documents by comparing the search tokens with the tokens in the index.

This approach has many advantages over the simple database search used by the regular search:

• Scoring of documents determine “how well” search hits match the terms. Documents with better scoring are sorted first
• Analyzers can use language specific “stemming” when generating the tokens. (E.g. the german words “Häuser”, “Hauses” and “Haus” will all generate the token “haus”). This improves the search experience, because searching for “Haus” will also find documents containing “Häuser” or “Hauses”.
• The german analyzer can be configured with a decompounder, which will create tokens for valid “subwords”. E.g. the german word “Weißkopfseeadler” will create the tokens “weiss”, “kopf”, “see” and “adler”. So when searching for “Adler”, documents containing “Weißkopfseeadler” will be found.
• In general, token based search in Elasticsearch works much faster than searching with wildcards in a database.