Connecting a Custom Data Source in Glean

This document outlines the process of integrating a custom data source in Glean. The process includes defining the source and permissions, syncing data, and validating results. Key prerequisites include source API access, understanding of objects to be indexed, and the permissions model. The process culminates with testing and monitoring to ensure seamless integration.

Step 1

Begin by identifying the data source, which will be Gainsight in this scenario. Determine the list of objects and fields that need to be synchronized with Glean.

Step 2

While identifying the objects and fields, ensure these are not already present in Glean groups or other data sources. Next, identify the list of users who should have access to this data. In Gainsight, two profiles have been identified: Full and Viewer Analytics.

Step 3

Assign access permissions to individuals with these two profiles, ensuring they have the requisite access to Glean's Gainsight data. After completing these identifications, proceed to create the custom data source in Glean by navigating to the Admin Console and selecting Data Sources.

Step 4

Add a new data source by choosing Custom Data Source and provide it with a unique name.

Step 5

Use the same name for the display name and select the appropriate category. Provide the instance URL of the data source.

Step 6

Ensure that you enable the option to use email addresses for document access control if required.

Step 7

Enable the necessary option for email address usage.

Step 8

Proceed by not creating object definitions in the UI, as this can be complex.

Step 9

Click on Publish to finalize the setup.

Step 10

Upon publishing, the data source will appear under the All Data Sources tab. Next, create an API indexing token for use with Glean APIs to retrieve data from Gainsight and push it to Glean. Click on Add Token and provide a meaningful description associated with the data source.

Step 11

Enable the option to create an indexing API token for use across all data sources.

Step 12

Specify that this API token is to be used exclusively with the Gainsight data source. Provide the required data source details, choose an expiry date, and click Save.

Step 13

Once saved, the API token will be generated. Ensure its secure storage and do not share it with others.

Step 14

After obtaining the API token, proceed to create object and property definitions for the data source based on the identified objects and fields from Gainsight.

Step 15

Use the Add Data Source Endpoint to see the payload with object and property definitions. Property definitions include a name, display name, and type of field, along with an option to hide the UI facet.

Step 16

Enable the field to be displayed as a filter in search results. Once enabled, it will appear as a filter option.

Step 17

Having set up the data source, verify that all properties are updated in the Glean data source.

Step 18

Verify in Glean that all object definitions and properties have synced.

Step 19

Verify in Gainsight that all pushed properties and objects have synced here as well.

Step 20

At this stage, everything should be configured in Glean.

Step 21

With the API indexing token and defined data source objects and properties, proceed to create middleware to sync data between Gainsight and Glean. Utilize Trailhead AI for automation.

Step 22

Define four key workflows: two for full loads and two for delta loads.

Step 23

Start with a full load workflow to index users into Glean by fetching user records from Gainsight matching the Full and Viewer Analytics profiles.

Step 24

Create a payload, establish a production group in Glean, and index users from Gainsight into Glean using the bulk index users endpoint. Add all indexed users to the group.

Step 25

After completing the full load, utilize a delta load workflow that identifies user changes every hour and updates the group accordingly.

Step 26

Once users and groups are set up, perform a full load with all identified records.

Step 27

Create a full load workflow to retrieve all object records and create a payload.

Step 28

Assign permissions to records based on the established group and index the documents using bulk index documents.

Step 29

Sync all records from Gainsight to Glean.

Step 30

Enable a delta load workflow to identify changes every hour and index them accordingly.

Step 31

Understand the difference between bulk index documents, which replaces previous loads, and index documents, used for incremental updates. Implement batching for large payloads and adhere to rate limits.

Step 32

Limit document indexing to 500 per batch with a 30-second delay to avoid rate limits.

Step 33

Test the results by querying the data source. For example, inquire about the last engagement date for 66degrees, and the system should retrieve the corresponding data.

Step 34

Perform a broader query, asking for specific customer details, and ensure the system provides accurate information from the Gainsight documents.

Step 35

The system should provide exact information and identify the documents where the data was fetched, completing the integration process. Thank you.