logo

pleroma

My custom branche(s) on git.pleroma.social/pleroma/pleroma git clone https://anongit.hacktivis.me/git/pleroma.git/

search.md (8117B)

    

  1. # Configuring search
  3. {! backend/administration/CLI_tasks/general_cli_task_info.include !}
  5. ## Built-in search
  7. To use built-in search that has no external dependencies, set the search module to `Pleroma.Activity`:
  9. > config :pleroma, Pleroma.Search, module: Pleroma.Search.DatabaseSearch
  11. While it has no external dependencies, it has problems with performance and relevancy.
  13. ## QdrantSearch
  15. This uses the vector search engine [Qdrant](https://qdrant.tech) to search the posts in a vector space. This needs a way to generate embeddings and uses the [OpenAI API](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings). This is implemented by several project besides OpenAI itself, including the python-based fastembed-server found in `supplemental/search/fastembed-api`.
  17. The default settings will support a setup where both the fastembed server and Qdrant run on the same system as pleroma. To use it, set the search provider and run the fastembed server, see the README in `supplemental/search/fastembed-api`:
  19. > config :pleroma, Pleroma.Search, module: Pleroma.Search.QdrantSearch
  21. Then, start the Qdrant server, see [here](https://qdrant.tech/documentation/quick-start/) for instructions.
  23. You will also need to create the Qdrant index once by running `mix pleroma.search.indexer create_index`. Running `mix pleroma.search.indexer index` will retroactively index the last 100_000 activities.
  25. ### Indexing and model options
  27. To see the available configuration options, check out the QdrantSearch section in `config/config.exs`.
  29. The default indexing option work for the default model (`snowflake-arctic-embed-xs`). To optimize for a low memory footprint, adjust the index configuration as described in the [Qdrant docs](https://qdrant.tech/documentation/guides/optimize/). See also [this blog post](https://qdrant.tech/articles/memory-consumption/) that goes into detail.
  31. Different embedding models will need different vector size settings. You can see a list of the models supported by the fastembed server [here](https://qdrant.github.io/fastembed/examples/Supported_Models), including their vector dimensions. These vector dimensions need to be set in the `qdrant_index_configuration`. 
  33. E.g, If you want to use `sentence-transformers/all-MiniLM-L6-v2` as a model, you will not need to adjust things, because it and `snowflake-arctic-embed-xs` are both 384 dimensional models. If you want to use `snowflake/snowflake-arctic-embed-l`, you will need to adjust the `size` parameter in the `qdrant_index_configuration` to 1024, as it has a dimension of 1024.
  35. When using a different model, you will need do drop the index and recreate it (`mix pleroma.search.indexer drop_index` and `mix pleroma.search.indexer create_index`), as the different embeddings are not compatible with each other.
  37. ## Meilisearch
  39. Note that it's quite a bit more memory hungry than PostgreSQL (around 4-5G for ~1.2 million
  40. posts while idle and up to 7G while indexing initially). The disk usage for this additional index is also
  41. around 4 gigabytes. Like [RUM](./cheatsheet.md#rum-indexing-for-full-text-search) indexes, it offers considerably
  42. higher performance and ordering by timestamp in a reasonable amount of time.
  43. Additionally, the search results seem to be more accurate.
  45. Due to high memory usage, it may be best to set it up on a different machine, if running pleroma on a low-resource
  46. computer, and use private key authentication to secure the remote search instance.
  48. To use [meilisearch](https://www.meilisearch.com/), set the search module to `Pleroma.Search.Meilisearch`:
  50. > config :pleroma, Pleroma.Search, module: Pleroma.Search.Meilisearch
  52. You then need to set the address of the meilisearch instance, and optionally the private key for authentication. You might
  53. also want to change the `initial_indexing_chunk_size` to be smaller if you're server is not very powerful, but not higher than `100_000`,
  54. because meilisearch will refuse to process it if it's too big. However, in general you want this to be as big as possible, because meilisearch
  55. indexes faster when it can process many posts in a single batch.
  57. > config :pleroma, Pleroma.Search.Meilisearch,
  58. >    url: "http://127.0.0.1:7700/",
  59. >    private_key: "private key",
  60. >    initial_indexing_chunk_size: 100_000
  62. Information about setting up meilisearch can be found in the
  63. [official documentation](https://docs.meilisearch.com/learn/getting_started/installation.html).
  64. You probably want to start it with `MEILI_NO_ANALYTICS=true` environment variable to disable analytics.
  65. At least version 0.25.0 is required, but you are strongly advised to use at least 0.26.0, as it introduces
  66. the `--enable-auto-batching` option which drastically improves performance. Without this option, the search
  67. is hardly usable on a somewhat big instance.
  69. ### Private key authentication (optional)
  71. To set the private key, use the `MEILI_MASTER_KEY` environment variable when starting. After setting the _master key_,
  72. you have to get the _private key_, which is actually used for authentication.
  74. === "OTP"
  75.     ```sh
  76.     ./bin/pleroma_ctl search.meilisearch show-keys <your master key here>
  77.     ```
  79. === "From Source"
  80.     ```sh
  81.     mix pleroma.search.meilisearch show-keys <your master key here>
  82.     ```
  84. You will see a "Default Admin API Key", this is the key you actually put into your configuration file.
  86. ### Initial indexing
  88. After setting up the configuration, you'll want to index all of your already existing posts. Only public posts are indexed.  You'll only
  89. have to do it one time, but it might take a while, depending on the amount of posts your instance has seen. This is also a fairly RAM
  90. consuming process for `meilisearch`, and it will take a lot of RAM when running if you have a lot of posts (seems to be around 5G for ~1.2
  91. million posts while idle and up to 7G while indexing initially, but your experience may be different).
  93. The sequence of actions is as follows:
  95. 1. First, change the configuration to use `Pleroma.Search.Meilisearch` as the search backend
  96. 2. Restart your instance, at this point it can be used while the search indexing is running, though search won't return anything
  97. 3. Start the initial indexing process (as described below with `index`),
  98.    and wait until the task says it sent everything from the database to index
  99. 4. Wait until everything is actually indexed (by checking with `stats` as described below),
  100.    at this point you don't have to do anything, just wait a while.
  102. To start the initial indexing, run the `index` command:
  104. === "OTP"
  105.     ```sh
  106.     ./bin/pleroma_ctl search.meilisearch index
  107.     ```
  109. === "From Source"
  110.     ```sh
  111.     mix pleroma.search.meilisearch index
  112.     ```
  114. This will show you the total amount of posts to index, and then show you the amount of posts indexed currently, until the numbers eventually
  115. become the same. The posts are indexed in big batches and meilisearch will take some time to actually index them, even after you have
  116. inserted all the posts into it. Depending on the amount of posts, this may be as long as several hours. To get information about the status
  117. of indexing and how many posts have actually been indexed, use the `stats` command:
  119. === "OTP"
  120.     ```sh
  121.     ./bin/pleroma_ctl search.meilisearch stats
  122.     ```
  124. === "From Source"
  125.     ```sh
  126.     mix pleroma.search.meilisearch stats
  127.     ```
  129. ### Clearing the index
  131. In case you need to clear the index (for example, to re-index from scratch, if that needs to happen for some reason), you can
  132. use the `clear` command:
  134. === "OTP"
  135.     ```sh
  136.     ./bin/pleroma_ctl search.meilisearch clear
  137.     ```
  139. === "From Source"
  140.     ```sh
  141.     mix pleroma.search.meilisearch clear
  142.     ```
  144. This will clear **all** the posts from the search index. Note, that deleted posts are also removed from index by the instance itself, so
  145. there is no need to actually clear the whole index, unless you want **all** of it gone. That said, the index does not hold any information
  146. that cannot be re-created from the database, it should also generally be a lot smaller than the size of your database. Still, the size
  147. depends on the amount of text in posts.