-
-
Notifications
You must be signed in to change notification settings - Fork 136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding feature descriptions #55
base: feature-descriptions
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,186 @@ | ||
# Federated Search | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's call this file federated-search.md so the url has the word search |
||
|
||
Federated or multi search is a way to search for documents in multiple collections as part of a single search query. You can also use multi-search to send multiple search queries to the same collections, essentially giving you a way to batch search queries in a single HTTP request. Federated search can help reduce network latencies. It can also be used to present similar content from other collections, that might encourage users to browse more content across your application. For example, your application might have different collections for `Nike` and `Adidas`. Now, if a user is looking for a shoe from a specific branch and they might not know what other brands are available, the search query can perform a search on both the collections and return relevant results from both the collections. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
to present results for the same search query from multiple collections... There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This example is a little confusing. Now that we have the BHP example below, let's remove this one. |
||
|
||
For example, if you search for `Canon` on https://www.bhphotovideo.com/, you would see that there are multiple results shown including products, suggestions and help resources: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
each of which could be indexed in separate collections, but the search term |
||
|
||
![bhp federated example](~@images/bhp-federated.png) | ||
|
||
Typesense supports searching across multiple collections in a single HTTP request. Let's create a search query for shoes: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could use make an example though would mimic the BHP example from above, searching across 3 collections? |
||
|
||
<Tabs :tabs="['JavaScript','PHP','Python','Ruby']"> | ||
<template v-slot:JavaScript> | ||
|
||
```javascript | ||
let searchRequests = { | ||
'searches': [ | ||
{ | ||
'collection': 'products', | ||
'q': 'shoe', | ||
'filter_by': 'price:=[50..120]' | ||
}, | ||
{ | ||
'collection': 'brands', | ||
'q': 'Nike' | ||
} | ||
] | ||
} | ||
|
||
// Search parameters that are common to all searches go here | ||
let commonSearchParams = { | ||
'query_by': 'name', | ||
} | ||
|
||
client.multiSearch.perform(searchRequests, commonSearchParams) | ||
``` | ||
</template> | ||
|
||
<template v-slot:PHP> | ||
|
||
```php | ||
$searchRequests = [ | ||
'searches' => [ | ||
[ | ||
'collection' => 'products', | ||
'q' => 'shoe', | ||
'filter_by' => 'price:=[50..120]' | ||
], | ||
[ | ||
'collection' => 'brands', | ||
'q' => 'Nike' | ||
] | ||
] | ||
]; | ||
|
||
// Search parameters that are common to all searches go here | ||
$commonSearchParams = [ | ||
'query_by' => 'name', | ||
]; | ||
|
||
$client->multiSearch->perform($searchRequests, $commonSearchParams); | ||
``` | ||
</template> | ||
<template v-slot:Python> | ||
|
||
```python | ||
search_requests = { | ||
'searches': [ | ||
{ | ||
'collection': 'products', | ||
'q': 'shoe', | ||
'filter_by': 'price:=[50..120]' | ||
}, | ||
{ | ||
'collection': 'brands', | ||
'q': 'Nike' | ||
} | ||
] | ||
} | ||
|
||
# Search parameters that are common to all searches go here | ||
common_search_params = { | ||
'query_by': 'name', | ||
} | ||
|
||
client.multi_search.perform(search_requests, common_search_params) | ||
``` | ||
</template> | ||
<template v-slot:Ruby> | ||
|
||
```ruby | ||
search_requests = { | ||
'searches': [ | ||
{ | ||
'collection': 'products', | ||
'q': 'shoe', | ||
'filter_by': 'price:=[50..120]' | ||
}, | ||
{ | ||
'collection': 'brands', | ||
'q': 'Nike' | ||
} | ||
] | ||
} | ||
|
||
# Search parameters that are common to all searches go here | ||
common_search_params = { | ||
'query_by': 'name', | ||
} | ||
|
||
client.multi_search.perform(search_requests, common_search_params) | ||
``` | ||
</template> | ||
</Tabs> | ||
|
||
Sample response: | ||
|
||
```json | ||
{ | ||
"results": [ | ||
{ | ||
"facet_counts": [], | ||
"found": 1, | ||
"hits": [ | ||
{ | ||
"document": { | ||
"name": "Blue shoe", | ||
"brand": "Adidas", | ||
"id": "126", | ||
"price": 50 | ||
}, | ||
"highlights": [ | ||
{ | ||
"field": "name", | ||
"matched_tokens": [ | ||
"shoe" | ||
], | ||
"snippet": "Blue <mark>shoe</mark>" | ||
} | ||
], | ||
"text_match": 130816 | ||
} | ||
], | ||
"out_of": 10, | ||
"page": 1, | ||
"request_params": { | ||
"per_page": 10, | ||
"q": "shoe" | ||
}, | ||
"search_time_ms": 1 | ||
}, | ||
{ | ||
"facet_counts": [], | ||
"found": 1, | ||
"hits": [ | ||
{ | ||
"document": { | ||
"name": "Nike shoes", | ||
"brand": "Nike", | ||
"id": "391", | ||
"price": 60 | ||
}, | ||
"highlights": [ | ||
{ | ||
"field": "name", | ||
"matched_tokens": [ | ||
"Nike" | ||
], | ||
"snippet": "<mark>Nike</mark>shoes" | ||
} | ||
], | ||
"text_match": 144112 | ||
} | ||
], | ||
"out_of": 5, | ||
"page": 1, | ||
"request_params": { | ||
"per_page": 10, | ||
"q": "Nike" | ||
}, | ||
"search_time_ms": 1 | ||
}, | ||
] | ||
} | ||
``` | ||
|
||
In the above example, the user is searching for a `Nike` shoe, but the `multiSerch` query returns results from the `Adidas` collection as well. You can control the number of maximum search requests using the `limit_multi_searches` parameter. By default, there is no limit. You can find more details on the argument [here](../../0.19.0/api/documents.html#federated-multi-search). |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,170 @@ | ||
# Scoped API Keys | ||
|
||
Typesense is designed with security and fine-grained access control in mind. To perform any action with Typesense, you need API keys. Typesense also allows access control on API keys. You can define capabilities as to what a user can or cannot do. You can also restrict access to a specific document or collection. In the case of a multi-tenant environment, you can scope API keys to a particular subset. This is helpful when you have indexed data from multiple tenants in your Typesense server and want to restrict users to only access their subset of data. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
... to a particular subset of records.
|
||
|
||
Typesense allows you to create API keys that have pre-defined filters embedded in them. So, whenever you run a search query with these API keys, those filters are automatically applied and cannot be overridden. You can then provide those search API keys to users and they would only be able to access the data that is allowed by the set filter. To create scoped API keys, you just need a parent key. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
... a parent key that only has |
||
|
||
Let's take example of a [company collection](../../api/collections.html#create-a-collection), that has the following documents: | ||
|
||
```shell | ||
{"company_id":124,"company_name":"Stark Industries","country":"USA","id":"0","num_employees":3355} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you update these records to contain one more field called For eg: {"accessible_by_user_id": [1, 2], "company_id":124,"company_name":"Stark Industries","country":"USA","id":"0","num_employees":3355}
... Then it's easier to explain with a real-world example: Let's say we now want to restrict users to only be able to access records that they have access to. We can create a scoped API key with an embedded or something along those lines |
||
{"company_id":125,"company_name":"Wayne Enterprises","country":"USA","id":"1","num_employees":4538} | ||
{"company_id":126,"company_name":"Daily Planet","country":"USA","id":"2","num_employees":2232} | ||
{"company_id":127,"company_name":"New Stark Industries","country":"USA","id":"3","num_employees":7945} | ||
``` | ||
|
||
Now, let's create a scoped API key that will restrict access to documents that have the `company_id` value as 124. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Before this, could you also add a code snippet to indicate how to create a parent key with just search permissions, so this article is self-contained? |
||
|
||
<Tabs :tabs="['JavaScript','PHP','Python','Ruby', 'Shell']"> | ||
<template v-slot:JavaScript> | ||
|
||
```javascript | ||
keyWithSearchPermissions = 'RN23GFr1s6jQ9kgSNg2O7fYcAUXU7127' | ||
client.keys().generateScopedSearchKey(keyWithSearchPermissions, {'filter_by': 'company_id:124', 'expires_at': 1611590465}) | ||
``` | ||
</template> | ||
|
||
<template v-slot:PHP> | ||
|
||
```php | ||
$keyWithSearchPermissions = 'RN23GFr1s6jQ9kgSNg2O7fYcAUXU7127'; | ||
$client->keys()->generateScopedSearchKey($keyWithSearchPermissions, ['filter_by' => 'company_id:124', 'expires_at' => 1611590465]); | ||
``` | ||
</template> | ||
<template v-slot:Python> | ||
|
||
```python | ||
key_with_search_permissions = 'RN23GFr1s6jQ9kgSNg2O7fYcAUXU7127' | ||
client.keys().generate_scoped_search_key(key_with_search_permissions, {"filter_by": "company_id:124", "expires_at": 1611590465}) | ||
``` | ||
</template> | ||
<template v-slot:Ruby> | ||
|
||
```ruby | ||
key_with_search_permissions = 'RN23GFr1s6jQ9kgSNg2O7fYcAUXU7127' | ||
client.keys().generate_scoped_search_key(key_with_search_permissions, {'filter_by': 'company_id:124', 'expires_at': 1611590465}) | ||
``` | ||
</template> | ||
<template v-slot:Shell> | ||
|
||
```bash | ||
KEY_WITH_SEARCH_PERMISSIONS="RN23GFr1s6jQ9kgSNg2O7fYcAUXU7127" | ||
EMBEDDED_SEARCH_PARAMETERS_JSON='{"filter_by":"company_id:124","expires_at":1611590465}' | ||
|
||
digest=$(echo -n $EMBEDDED_SEARCH_PARAMETERS_JSON | openssl dgst -sha256 -hmac $KEY_WITH_SEARCH_PERMISSIONS -binary | base64) | ||
|
||
scoped_api_key=$(echo -n "${digest}${KEY_WITH_SEARCH_PERMISSIONS:0:4}${EMBEDDED_SEARCH_PARAMETERS_JSON}" | base64) | ||
|
||
echo $scoped_api_key | ||
``` | ||
|
||
</template> | ||
</Tabs> | ||
|
||
Sample response: | ||
|
||
```json | ||
"RDhxa2VKTnBQVkxaVlFIOS9JWDZ2bDdtMU5HL3laa0pab2pTeEUzbFBhZz1STjIzeyJmaWx0ZXJfYnkiOiJjb21wYW55X2lkOjEyNCIsImV4cGlyZXNfYXQiOjE2MTE1OTA0NjV9" | ||
``` | ||
|
||
The `expires_at` parameter sets the expiration date for the API key and must be less that the expiration of parent API key. Let's perform a search using the scoped API key: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. New paragraph for "Let's perform a search using the scoped API key" |
||
|
||
<Tabs :tabs="['JavaScript','PHP','Python','Ruby']"> | ||
<template v-slot:JavaScript> | ||
|
||
```javascript | ||
let searchParameters = { | ||
'q' : 'Stark', | ||
'query_by' : 'company_name', | ||
'sort_by' : 'num_employees:desc' | ||
} | ||
|
||
client.collections('companies') | ||
.documents() | ||
.search(searchParameters) | ||
.then(function (searchResults) { | ||
console.log(searchResults) | ||
}) | ||
``` | ||
</template> | ||
|
||
<template v-slot:PHP> | ||
|
||
```php | ||
$$searchParameters = [ | ||
'q' => 'Stark', | ||
'query_by' => 'company_name', | ||
'sort_by' => 'num_employees:desc' | ||
] | ||
|
||
$client->collections['companies']->documents->search($searchParameters) | ||
``` | ||
</template> | ||
<template v-slot:Python> | ||
|
||
```python | ||
search_parameters = { | ||
'q' : 'Stark', | ||
'query_by' : 'company_name', | ||
'sort_by' : 'num_employees:desc' | ||
} | ||
|
||
client.collections['companies'].documents.search(search_parameters) | ||
``` | ||
</template> | ||
<template v-slot:Ruby> | ||
|
||
```ruby | ||
search_parameters = { | ||
'q' => 'Stark', | ||
'query_by' => 'company_name', | ||
'sort_by' => 'num_employees:desc' | ||
} | ||
|
||
client.collections['companies'].documents.search(search_parameters) | ||
``` | ||
</template> | ||
</Tabs> | ||
|
||
|
||
Response: | ||
|
||
```json | ||
{ | ||
"facet_counts": [], | ||
"found": 1, | ||
"hits": [ | ||
{ | ||
"document": { | ||
"company_id": 124, | ||
"company_name": "Stark Industries", | ||
"country": "USA", | ||
"id": "0", | ||
"num_employees": 3355 | ||
}, | ||
"highlights": [ | ||
{ | ||
"field": "company_name", | ||
"matched_tokens": [ | ||
"Stark" | ||
], | ||
"snippet": "<mark>Stark</mark> Industries" | ||
} | ||
], | ||
"text_match": 130816 | ||
} | ||
], | ||
"out_of": 4, | ||
"page": 1, | ||
"request_params": { | ||
"collection_name": "companies", | ||
"per_page": 10, | ||
"q": "stark" | ||
}, | ||
"search_time_ms": 0 | ||
} | ||
``` | ||
|
||
As you see in the response, the document with `company_id` set to 124 is shown in the output. There is another document that has the `company_name` as "New Stark Industries", but it won't be shown in the result. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
... because it does not match the filter |
||
|
||
You can find more details about scoped API keys [here](../../api/api-keys.html#generate-scoped-search-key). |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Raft Based Clustering | ||
|
||
High availability is essential for production environments. Typesense uses the [Raft Consensus Algorithm](https://raft.github.io/) to create a highly available cluster with more than one Typesense servers. With Raft, you need to create a cluster of 3 nodes to tolerate single node failures. If you wish to handle 2-node failures, then you need a minimum of 5 nodes in the cluster. Note that adding more nodes will also increase write latencies. | ||
|
||
More details on cluster operations can be found [here](../../api/cluster-operations.html). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I run this on dev, I see this error:
I don't see a file called curation.md in the repo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't added it yet. Will be added in future PR. Do you want me to add a placeholder file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I see, in that case let's remove the line from the side-bar configuration, so the nav renders in the meantime.