> ## Documentation Index
> Fetch the complete documentation index at: https://cortex-e852fafe-auto-update-openapi-6a9e3873a7492d091ac8c1f.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Build your own Glean with HydraDB

> Learn how to build a comprehensive workplace search and AI assistant platform using HydraDB APIs. This guide covers data ingestion, retrieval, and app-layer answer generation across multiple data sources.

This guide will walk you through building an extremely powerful workplace search and AI assistant platform that rivals Glean using HydraDB APIs. You'll learn how to create a unified retrieval experience across multiple data sources and generate answers in your application layer.

> **Note**: All code in this guide uses the official HydraDB Python SDK. Base URL: `https://api.hydradb.com`. Get your API key at [app.hydradb.com](https://app.hydradb.com).

## Prerequisites

**Required knowledge**: Python basics, REST APIs, environment variables\
**Required tools**:

* HydraDB API key
* Python 3.11 or 3.12 (`python --version`)
* `pip install hydradb-sdk`

## What You'll Build

By the end of this cookbook, you'll be able to:

* Ingest documents from multiple data sources (Slack, email, Google Drive, Jira) into a unified HydraDB knowledge base
* Run natural language workplace search across all sources with a single [`/query`](/api-reference/v2/endpoint/query) call
* Personalize search results per user by storing and retrieving AI memories
* Generate grounded AI answers from retrieved context using any LLM

## Overview

A Glean-like application typically includes these core features:

* **Universal Search**: Search across multiple data sources (documents, emails, chats, etc.)
* **Retrieval-Assisted Answers**: Generate intelligent answers from retrieved company knowledge
* **AI Memories for User Preferences**: Remember and adapt to individual user preferences, search patterns, and behavioral patterns
* **Data Ingestion**: Connect to various apps and services
* **Knowledge Graph**: Build connections between information
* **Security & Access Control**: Role-based permissions and data isolation

## Architecture Overview

```mermaid theme={null}
graph TD
    A["Frontend UI<br/>• Search UI<br/>• Chat Interface<br/>• Results View"] 
    B["Backend API<br/>• Data Sync<br/>• Auth/ACL<br/>• App Connectors"]
    C["HydraDB APIs<br/>• Retrieval Engine<br/>• Document Index<br/>• Memory Store"]
    
    D["Data Sources<br/>• Google apps<br/>• Slack<br/>• Notion<br/>• Jira"]
    E["App Connectors for Fetching Data<br/><br/>• Composio<br/>• Vanilla APIs<br/>• Webhooks<br/>• Scheduled Jobs"]
    F["Retrieval Layer<br/>• HydraDB Memory<br/>• User Sessions<br/>• Metadata Store"]
    
    A <--> B
    B <--> C
    A --> D
    B <--> E
    C --> F
```

## Step 1: Data Ingestion Strategy

### 1.1 App Connectors Setup

You'll need to connect to various data sources. Here are recommended approaches:

#### Option A: Using Composio

[Composio](https://composio.dev/) provides pre-built connectors for popular apps:

```javascript theme={null}
// Example: Setting up Composio connector
const composioConfig = {
  connectors: [
    {
      name: 'slack',
      config: {
        token: process.env.SLACK_BOT_TOKEN,
        channels: ['general', 'random', 'project-*']
      }
    },
    {
      name: 'gmail',
      config: {
        credentials: process.env.GMAIL_CREDENTIALS,
        labels: ['INBOX', 'SENT', 'IMPORTANT']
      }
    },
    {
      name: 'notion',
      config: {
        token: process.env.NOTION_TOKEN,
        databases: ['projects', 'docs', 'meetings']
      }
    }
  ]
};
```

#### Option B: Vanilla API Integration

For custom integrations, use the native APIs:

```javascript theme={null}
// Example: Slack API integration
class SlackConnector {
  constructor(token) {
    this.token = token;
    this.client = new WebClient(token);
  }

  async fetchMessages(channelId, limit = 100) {
    const result = await this.client.conversations.history({
      channel: channelId,
      limit: limit
    });
    
    return result.messages.map(msg => ({
      id: msg.ts,
      text: msg.text,
      user: msg.user,
      timestamp: msg.ts,
      channel: channelId,
      type: 'slack_message'
    }));
  }

  async fetchChannels() {
    const result = await this.client.conversations.list();
    return result.channels;
  }
}

// Example: Gmail API integration
class GmailConnector {
  constructor(credentials) {
    this.gmail = google.gmail({ version: 'v1', auth: credentials });
  }

  async fetchEmails(query = 'in:inbox', maxResults = 100) {
    const response = await this.gmail.users.messages.list({
      userId: 'me',
      q: query,
      maxResults: maxResults
    });

    const emails = [];
    for (const message of response.data.messages) {
      const email = await this.gmail.users.messages.get({
        userId: 'me',
        id: message.id
      });
      
      emails.push({
        id: email.data.id,
        subject: this.getHeader(email.data.payload.headers, 'Subject'),
        from: this.getHeader(email.data.payload.headers, 'From'),
        body: this.getBody(email.data.payload),
        timestamp: email.data.internalDate,
        type: 'gmail'
      });
    }
    
    return emails;
  }
}
```

### 1.2 Data Normalization

Create a unified data format for all sources:

> **Important**: For optimal performance, limit each batch to a maximum of **20 app sources** per request. Send multiple batch requests with an interval of **1 second** between each request.

```javascript theme={null}
// Unified data structure for HydraDB app upload
const normalizedData = {
  id: 'unique_id',
  database: 'your_database',
  collection: 'your_collection',
  title: 'Document/Message Title',
  type: 'slack_message', // Source category: gmail, slack_message, notion_page, document, etc.
  timestamp: '2024-01-01T00:00:00Z', // ISO timestamp
  content: {
    text: 'Main content text',
    html_base64: 'base64_encoded_html',
    markdown: 'markdown_content'
  },
  url: 'https://app.com/item/123', // Optional: source URL
  description: 'Optional description of the source', // Optional
  metadata: {}, // Optional database-level metadata
  additional_metadata: {
    author: 'user@company.com',
    id: 'original_id',
    tags: ['project-a', 'urgent', 'meeting-notes'],
    permissions: ['user1@company.com', 'user2@company.com']
  }
};
```

### 1.3 Batch Upload to HydraDB

Use HydraDB's batch upload capabilities for efficient data ingestion:

> **Best Practice**: Always verify processing after upload using the `/context/status` endpoint to ensure your data is properly indexed.

<CodeGroup>
  ```python Python SDK theme={null}
  import os
  import json
  from hydra_db import HydraDB

  client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

  # Upload a batch of knowledge sources
  def upload_batch(sources: list, database: str, collection: str = None):
      app_knowledge = [
          {**source, "database": database, "collection": collection or database}
          for source in sources
      ]

      result = client.context.ingest(
          database=database,
          app_knowledge=json.dumps(app_knowledge)
      )
      return result

  # Upload with verification — confirm each item is indexed before proceeding
  def upload_with_verification(sources: list, database: str, collection: str = None):
      upload_result = upload_batch(sources, database, collection)

      if upload_result.data.results:
          for item in upload_result.data.results:
              id = item.id
              status = client.context.status(
                  database=database,
                  ids=[id]
              )
              items = status.data.statuses or []
              if items and items[0].indexing_status == "errored":
                  raise Exception(f"Processing failed for source {id}")

      return upload_result
  ```

  ```typescript TypeScript SDK theme={null}
  import { HydraDBClient } from "@hydradb/sdk";

  const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

  // Upload a batch of knowledge sources
  const uploadBatch = async (sources: any[], database: string, collection?: string) => {
    const appKnowledge = sources.map(source => ({
      ...source,
      database: database,
      collection: collection || database
    }));

    const result = await client.context.ingest({
      database: database,
      appKnowledge: JSON.stringify(appKnowledge)
    });

    return result;
  };

  // Upload with verification — confirm each item is indexed before proceeding
  const uploadWithVerification = async (sources: any[], database: string, collection?: string) => {
    const uploadResult = await uploadBatch(sources, database, collection);

    const items = uploadResult.data?.results ?? [];
    for (const item of items) {
      const id = item.id;
      const status = await client.context.status({
        database: database,
        ids: [id]
      });
      const statusItem = status.data.statuses[0];
      if (statusItem?.indexingStatus === "errored") {
        throw new Error(`Processing failed for source ${id}`);
      }
    }

    return uploadResult;
  };
  ```
</CodeGroup>

## Step 2: Search and Answer Generation

### 2.1 Universal Search Interface

Create a search interface that queries across all data sources:

> **Note**: HydraDB supports filtering by `source_title` and `source_type` using the `metadata` parameter. Use these for targeted searches across specific data sources.

<CodeGroup>
  ```python Python SDK theme={null}
  import os
  from hydra_db import HydraDB

  client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

  def search(query: str, database: str, collection: str = None,
             max_results: int = 10, mode: str = "fast",
             metadata: dict = None):
      return client.query(
          query=query,
          database=database,
          collection=collection,
          max_results=max_results,
          mode=mode,
          alpha=0.5,        # Balance semantic vs keyword search (0.0 to 1.0)
          recency_bias=0.3, # Recency preference (0.0 to 1.0)
          **({"metadata": metadata} if metadata else {}),
      )

  # Filter search by source type
  def query_by_source_type(query: str, database: str, source_type: str):
      return search(query, database, metadata={"source_type": source_type})

  # Filter search by source title
  def query_by_source_title(query: str, database: str, source_title: str):
      return search(query, database, metadata={"source_title": source_title})
  ```

  ```typescript TypeScript SDK theme={null}
  import { HydraDBClient } from "@hydradb/sdk";

  const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

  interface SearchOptions {
    collection?: string;
    max_results?: number;
    mode?: string;
    metadata?: Record<string, any>;
  }

  const search = async (query: string, database: string, options: SearchOptions = {}) => {
    const {
      collection,
      max_results = 10,
      mode = "fast",
      metadata,
    } = options;

    return await client.query({
      query,
      database: database,
      collection: collection,
      maxResults: max_results,
      mode,
      alpha: 0.5,         // Balance semantic vs keyword search (0.0 to 1.0)
      recencyBias: 0.3,  // Recency preference (0.0 to 1.0)
      ...(metadata && { metadata }),
    });
  };

  // Filter search by source type
  const searchBySourceType = (query: string, database: string, sourceType: string) =>
    search(query, database, { metadata: { source_type: sourceType } });

  // Filter search by source title
  const searchBySourceTitle = (query: string, database: string, sourceTitle: string) =>
    search(query, database, { metadata: { source_title: sourceTitle } });
  ```
</CodeGroup>

### 2.2 Advanced Search Features

Implement advanced search capabilities:

> **Advanced Features**:
>
> * `mode`: Use `"thinking"` for multi-query retrieval with reranking, or `"fast"` for single-query retrieval
> * `alpha`: Controls semantic vs keyword matching (0.0-1.0, or `"auto"`)
> * `recency_bias`: Prioritizes recent content (0.0-1.0)

<CodeGroup>
  ```python Python SDK theme={null}
  import os
  import json
  from hydra_db import HydraDB

  client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

  # Search with optional source type / title metadata filters
  def search_with_filters(query: str, database: str, source_types: str = None, source_titles: str = None):
      metadata = {}
      if source_types:
          metadata["source_type"] = source_types
      if source_titles:
          metadata["source_title"] = source_titles

      return client.query(
          query=query,
          database=database,
          max_results=10,
          mode="fast",
          alpha=0.5,
          recency_bias=0.3,
          **({"metadata": metadata} if metadata else {})
      )

  # Guide retrieval with additional context — prepend context to the query string
  def search_with_context(query: str, database: str, context: str):
      return client.query(
          query=f"{context}\n\n{query}" if context else query,
          database=database,
          max_results=10,
          mode="fast",
          alpha=0.5,
          recency_bias=0.3,
      )

  # Conversational search — fold prior conversation turns into the query
  def conversational_search(query: str, database: str, conversation_history: list = None):
      history = conversation_history or []
      enriched_query = (
          f"Previous conversation context: {json.dumps(history)}\n\n{query}"
          if history else query
      )
      return client.query(
          query=enriched_query,
          database=database,
          max_results=10,
          mode="fast",
          alpha=0.5,
          recency_bias=0.3,
      )
  ```

  ```typescript TypeScript SDK theme={null}
  import { HydraDBClient } from "@hydradb/sdk";

  const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

  // Search with optional source type / title metadata filters
  const searchWithFilters = async (
    query: string,
    database: string,
    filters: { sourceTypes?: string; sourceTitles?: string } = {}
  ) => {
    const metadata: Record<string, any> = {};
    if (filters.sourceTypes) metadata.source_type = filters.sourceTypes;
    if (filters.sourceTitles) metadata.source_title = filters.sourceTitles;

    return await client.query({
      query,
      database: database,
      maxResults: 10,
      mode: "fast",
      alpha: 0.5,
      recencyBias: 0.3,
      ...(Object.keys(metadata).length > 0 && { metadata })
    });
  };

  // Guide retrieval with additional context — prepend context to the query string
  const searchWithContext = async (query: string, database: string, context: string) =>
    client.query({
      query: context ? `${context}\n\n${query}` : query,
      database: database,
      maxResults: 10,
      mode: "fast",
      alpha: 0.5,
      recencyBias: 0.3,
    });

  // Conversational search — fold prior conversation turns into the query
  const conversationalSearch = async (
    query: string,
    database: string,
    conversationHistory: object[] = []
  ) =>
    client.query({
      query: conversationHistory.length
        ? `Previous conversation context: ${JSON.stringify(conversationHistory)}\n\n${query}`
        : query,
      database: database,
      maxResults: 10,
      mode: "fast",
      alpha: 0.5,
      recencyBias: 0.3,
    });
  ```
</CodeGroup>

### 2.3 AI Memories and User Preferences

One of the most powerful features of building a Glean-like application with HydraDB is leveraging **AI Memories** to create truly personalized experiences. HydraDB automatically manages AI memories using `collection` for user-level isolation. This allows your application to remember user preferences, past interactions, and behavioral patterns, making every search and interaction more relevant and efficient.

#### Understanding AI Memories

HydraDB's AI memories are dynamic, user-specific profiles that evolve over time. They capture not just what users say, but their intentions, preferences, and unique behaviors. HydraDB automatically manages these memories using `collection` for user-level isolation. This enables your Glean clone to:

* **Remember User Preferences**: Format preferences, source preferences, search patterns
* **Understand Intent**: Learn what types of information users typically seek
* **Adapt Responses**: Tailor answers based on past interactions
* **Anticipate Needs**: Suggest relevant information before users ask

#### Implementing AI Memories in Your Search

<CodeGroup>
  ```python Python SDK theme={null}
  import os
  from datetime import datetime
  from hydra_db import HydraDB

  client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

  # In-memory profile store — replace with your own database in production
  user_profiles: dict = {}

  def get_user_profile(user_id: str) -> dict:
      if user_id not in user_profiles:
          user_profiles[user_id] = {
              "user_id": user_id,
              "preferred_mode": "fast",
              "preferred_source_types": [],
              "frequent_queries": [],
              "preferred_formats": ["bullet_points"],
              "response_style": "concise",
              "favorite_sources": [],
              "search_history": [],
              "last_interaction": None
          }
      return user_profiles[user_id]

  def build_personalized_instructions(profile: dict, query: str) -> str:
      instructions = "User preferences: "
      if "bullet_points" in profile["preferred_formats"]:
          instructions += "Prefer bullet point responses. "
      if profile["response_style"] == "concise":
          instructions += "Keep responses concise and to the point. "
      if profile["favorite_sources"]:
          instructions += f"User frequently uses sources: {', '.join(profile['favorite_sources'])}. "
      if profile["frequent_queries"]:
          instructions += f"User often searches for: {', '.join(profile['frequent_queries'][:3])}. "
      instructions += f"Current query: {query}"
      return instructions

  def save_user_profile(user_id: str, profile: dict):
      # Persist to your backend database
      print(f"Saving profile for user {user_id}:", profile)

  # Search with personalized query enrichment derived from the user's local profile
  def search_with_memory(query: str, database: str, user_id: str):
      profile = get_user_profile(user_id)
      enriched_query = f"{build_personalized_instructions(profile, query)}\n\n{query}"

      kwargs = dict(
          query=enriched_query,
          database=database,
          collection=user_id,
          max_results=10,
          mode=profile["preferred_mode"] or "fast",
          alpha=0.5,
          recency_bias=0.3,
      )
      if profile["preferred_source_types"]:
          kwargs["metadata"] = {"source_type": profile["preferred_source_types"]}

      search_results = client.query(**kwargs)

      # Update local profile history
      profile["search_history"].append({
          "query": query,
          "timestamp": datetime.utcnow().isoformat(),
          "result_count": len(search_results.data.chunks or [])
      })
      profile["search_history"] = profile["search_history"][-50:]

      if not any(query.lower() in q.lower() for q in profile["frequent_queries"]):
          profile["frequent_queries"] = ([query] + profile["frequent_queries"])[:10]

      for chunk in (search_results.data.chunks or []):
          if chunk.source_title and chunk.source_title not in profile["favorite_sources"]:
              profile["favorite_sources"].append(chunk.source_title)
      profile["favorite_sources"] = profile["favorite_sources"][:5]

      profile["last_interaction"] = datetime.utcnow().isoformat()
      save_user_profile(user_id, profile)

      return search_results
  ```

  ```typescript TypeScript SDK theme={null}
  import { HydraDBClient } from "@hydradb/sdk";

  const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

  // In-memory profile store — replace with your own database in production
  const userProfiles = new Map<string, any>();

  const getUserProfile = (userId: string) => {
    if (!userProfiles.has(userId)) {
      userProfiles.set(userId, {
        userId,
        preferredMode: "fast",
        preferredSourceTypes: [] as string[],
        frequentQueries: [] as string[],
        preferredFormats: ["bullet_points"],
        responseStyle: "concise",
        favoriteSources: [] as string[],
        searchHistory: [] as object[],
        lastInteraction: null as string | null
      });
    }
    return userProfiles.get(userId)!;
  };

  const buildPersonalizedInstructions = (userProfile: any, query: string): string => {
    let instructions = "User preferences: ";
    if (userProfile.preferredFormats.includes("bullet_points")) instructions += "Prefer bullet point responses. ";
    if (userProfile.responseStyle === "concise") instructions += "Keep responses concise and to the point. ";
    if (userProfile.favoriteSources.length > 0) instructions += `User frequently uses sources: ${userProfile.favoriteSources.join(", ")}. `;
    if (userProfile.frequentQueries.length > 0) instructions += `User often searches for: ${userProfile.frequentQueries.slice(0, 3).join(", ")}. `;
    instructions += `Current query: ${query}`;
    return instructions;
  };

  const saveUserProfile = async (userId: string, profile: any) => {
    // Persist to your backend database
    console.log(`Saving profile for user ${userId}:`, profile);
  };

  // Search with personalized query enrichment derived from the user's local profile
  const searchWithMemory = async (query: string, database: string, userId: string) => {
    const userProfile = getUserProfile(userId);
    const enrichedQuery = `${buildPersonalizedInstructions(userProfile, query)}\n\n${query}`;

    const searchResults = await client.query({
      query: enrichedQuery,
      database: database,
      collection: userId,
      maxResults: 10,
      mode: userProfile.preferredMode || "fast",
      alpha: 0.5,
      recencyBias: 0.3,
      ...(userProfile.preferredSourceTypes?.length > 0 && {
        metadata: { source_type: userProfile.preferredSourceTypes }
      })
    });

    // Update local profile history
    userProfile.searchHistory.push({ query, timestamp: new Date().toISOString(), resultCount: searchResults.data.chunks?.length || 0 });
    if (userProfile.searchHistory.length > 50) userProfile.searchHistory = userProfile.searchHistory.slice(-50);

    const queryLower = query.toLowerCase();
    if (!userProfile.frequentQueries.some((q: string) => q.toLowerCase().includes(queryLower))) {
      userProfile.frequentQueries = [query, ...userProfile.frequentQueries].slice(0, 10);
    }

    if (searchResults.data.chunks) {
      for (const chunk of searchResults.data.chunks) {
        if (chunk.source && !userProfile.favoriteSources.includes(chunk.source)) {
          userProfile.favoriteSources.push(chunk.source);
        }
      }
      userProfile.favoriteSources = userProfile.favoriteSources.slice(0, 5);
    }

    userProfile.lastInteraction = new Date().toISOString();
    await saveUserProfile(userId, userProfile);

    return searchResults;
  };
  ```
</CodeGroup>

#### User Preference Learning

```javascript theme={null}
class PreferenceLearner {
  constructor() {
    this.preferencePatterns = new Map();
  }

  async learnFromInteraction(userId, interaction) {
    const {
      query,
      selectedResults,
      responseFormat,
      searchFilters,
      timeSpent,
      followUpQueries
    } = interaction;

    const patterns = this.preferencePatterns.get(userId) || {
      queryPatterns: [],
      formatPreferences: {},
      sourcePreferences: {},
      timePatterns: [],
      filterPreferences: {}
    };

    // Learn query patterns
    this.learnQueryPatterns(patterns, query, followUpQueries);
    
    // Learn format preferences
    this.learnFormatPreferences(patterns, responseFormat, selectedResults);
    
    // Learn source preferences
    this.learnSourcePreferences(patterns, selectedResults);
    
    // Learn time patterns
    this.learnTimePatterns(patterns, timeSpent);
    
    // Learn filter preferences
    this.learnFilterPreferences(patterns, searchFilters);

    this.preferencePatterns.set(userId, patterns);
  }

  learnQueryPatterns(patterns, query, followUpQueries) {
    // Analyze query complexity, length, and type
    const queryAnalysis = {
      length: query.length,
      complexity: this.analyzeComplexity(query),
      type: this.classifyQueryType(query),
      hasFilters: query.includes('in:') || query.includes('from:'),
      timestamp: new Date().toISOString()
    };

    patterns.queryPatterns.push(queryAnalysis);
    
    // Keep only recent patterns
    if (patterns.queryPatterns.length > 100) {
      patterns.queryPatterns = patterns.queryPatterns.slice(-100);
    }
  }

  learnFormatPreferences(patterns, format, selectedResults) {
    if (!patterns.formatPreferences[format]) {
      patterns.formatPreferences[format] = 0;
    }
    patterns.formatPreferences[format]++;
  }

  learnSourcePreferences(patterns, selectedResults) {
    selectedResults.forEach(result => {
      const sourceType = result.source;
      if (!patterns.sourcePreferences[sourceType]) {
        patterns.sourcePreferences[sourceType] = 0;
      }
      patterns.sourcePreferences[sourceType]++;
    });
  }

  analyzeComplexity(query) {
    const words = query.split(' ').length;
    const hasQuotes = query.includes('"');
    const hasOperators = /AND|OR|NOT|in:|from:|to:/.test(query);
    
    let complexity = 'simple';
    if (words > 5 || hasQuotes || hasOperators) complexity = 'complex';
    if (words > 10 || (hasQuotes && hasOperators)) complexity = 'advanced';
    
    return complexity;
  }

  classifyQueryType(query) {
    if (query.includes('how to') || query.includes('steps')) return 'how-to';
    if (query.includes('what is') || query.includes('define')) return 'definition';
    if (query.includes('when') || query.includes('schedule')) return 'temporal';
    if (query.includes('who') || query.includes('contact')) return 'person';
    if (query.includes('where') || query.includes('location')) return 'location';
    return 'general';
  }
}
```

#### Memory-Enhanced Search Interface

```javascript theme={null}
const MemoryEnhancedSearch = ({ userId }) => {
  const [query, setQuery] = useState('');
  const [results, setResults] = useState(null);
  const [userPreferences, setUserPreferences] = useState(null);
  const [suggestions, setSuggestions] = useState([]);

  const searchClient = new PersonalizedSearch(API_KEY, TENANT_ID);
  const preferenceLearner = new PreferenceLearner();

  useEffect(() => {
    // Load user preferences on component mount
    loadUserPreferences();
  }, [userId]);

  const loadUserPreferences = async () => {
    const profile = await searchClient.getUserProfile(userId);
    setUserPreferences(profile);
    
    // Generate search suggestions based on user's history
    const suggestions = generateSuggestions(profile);
    setSuggestions(suggestions);
  };

  const generateSuggestions = (profile) => {
    const suggestions = [];
    
    // Suggest based on frequent queries
    if (profile.frequentQueries.length > 0) {
      suggestions.push({
        type: 'frequent',
        text: profile.frequentQueries[0],
        label: 'Frequently searched'
      });
    }
    
    // Suggest based on recent searches
    if (profile.searchHistory.length > 0) {
      const recentSearches = profile.searchHistory
        .slice(-3)
        .map(h => h.query);
      
      recentSearches.forEach(query => {
        suggestions.push({
          type: 'recent',
          text: query,
          label: 'Recent search'
        });
      });
    }
    
    return suggestions;
  };

  const handleSearch = async () => {
    const startTime = Date.now();
    
    try {
      const searchResults = await searchClient.searchWithMemory(query, userId);
      setResults(searchResults);
      
      // Learn from this interaction
      const interaction = {
        query,
        selectedResults: [], // Will be populated when user clicks results
        responseFormat: searchResults.format || 'default',
        searchFilters: {},
        timeSpent: Date.now() - startTime,
        followUpQueries: []
      };
      
      await preferenceLearner.learnFromInteraction(userId, interaction);
      
    } catch (error) {
      console.error('Search failed:', error);
    }
  };

  const handleResultClick = async (result) => {
    // Update user preferences when they click on results
    const profile = await searchClient.getUserProfile(userId);
    
    // Mark this source type as preferred
    if (result.source && !profile.preferredSourceTypes.includes(result.source)) {
      profile.preferredSourceTypes.push(result.source);
      await searchClient.saveUserProfile(userId, profile);
    }
  };

  return (
    <div className="memory-enhanced-search">
      <div className="search-header">
        <input
          type="text"
          value={query}
          onChange={(e) => setQuery(e.target.value)}
          placeholder="Search with your preferences in mind..."
          className="search-input"
        />
        <button onClick={handleSearch}>Search</button>
      </div>

      {/* Show personalized suggestions */}
      {suggestions.length > 0 && (
        <div className="search-suggestions">
          <h4>Based on your preferences:</h4>
          {suggestions.map((suggestion, index) => (
            <button
              key={index}
              className="suggestion-chip"
              onClick={() => setQuery(suggestion.text)}
            >
              {suggestion.text}
              <span className="suggestion-label">{suggestion.label}</span>
            </button>
          ))}
        </div>
      )}

      {/* Show user preferences summary */}
      {userPreferences && (
        <div className="user-preferences">
          <details>
            <summary>Your Search Preferences</summary>
            <div className="preferences-content">
              <p><strong>Preferred sources:</strong> {userPreferences.preferredSourceTypes.join(', ') || 'None set'}</p>
              <p><strong>Response style:</strong> {userPreferences.responseStyle}</p>
              <p><strong>Frequent searches:</strong> {userPreferences.frequentQueries.slice(0, 3).join(', ')}</p>
            </div>
          </details>
        </div>
      )}

      {results && (
        <SearchResults 
          results={results} 
          onResultClick={handleResultClick}
          userPreferences={userPreferences}
        />
      )}
    </div>
  );
};
```

#### Benefits of AI Memories in Your Glean Clone

1. **Personalized Search Results**: Users get results tailored to their preferences and past behavior
2. **Faster Information Discovery**: The system learns what sources and formats users prefer
3. **Improved User Experience**: Every interaction feels more personal and relevant
4. **Reduced Cognitive Load**: Users don't need to repeat their preferences or search patterns
5. **Adaptive Learning**: The system continuously improves based on user interactions
6. **Automatic Management**: HydraDB handles memory updates automatically - no manual implementation required

#### Best Practices for AI Memories

* **Respect Privacy**: Always give users control over their data and preferences
* **Transparency**: Show users what preferences are being used and allow them to modify them
* **Graceful Degradation**: Ensure the system works well even without user history
* **Collection Isolation**: Use `collection` to isolate user data for personalized experiences
* **Performance**: Cache user profiles to avoid repeated API calls

## Step 3: Frontend Implementation

### 3.1 Search Interface

```javascript theme={null}
// React component example
import React, { useState, useEffect } from 'react';

const GleanSearchInterface = () => {
  const [query, setQuery] = useState('');
  const [results, setResults] = useState(null);
  const [loading, setLoading] = useState(false);
  const [filters, setFilters] = useState({
    sourceTypes: [],
    dateRange: null,
    authors: []
  });

  const searchClient = new AdvancedSearch(API_KEY, TENANT_ID);

  const handleSearch = async () => {
    setLoading(true);
    try {
      const searchResults = await searchClient.searchWithFilters(query, filters);
      setResults(searchResults);
    } catch (error) {
      console.error('Search failed:', error);
    } finally {
      setLoading(false);
    }
  };

  return (
    <div className="glean-search">
      <div className="search-header">
        <input
          type="text"
          value={query}
          onChange={(e) => setQuery(e.target.value)}
          placeholder="Search across all your work..."
          className="search-input"
        />
        <button onClick={handleSearch} disabled={loading}>
          {loading ? 'Searching...' : 'Search'}
        </button>
      </div>

      <div className="search-filters">
        <SourceTypeFilter
          value={filters.sourceTypes}
          onChange={(types) => setFilters({...filters, sourceTypes: types})}
        />
        <DateRangeFilter
          value={filters.dateRange}
          onChange={(range) => setFilters({...filters, dateRange: range})}
        />
      </div>

      {results && (
        <SearchResults results={results} />
      )}
    </div>
  );
};
```

### 3.2 Results Display

```javascript theme={null}
const SearchResults = ({ results }) => {
  const { chunks, graphContext } = results.data ?? {};

  return (
    <div className="search-results">
      {chunks && chunks.length > 0 && (
        <div className="source-results">
          <h3>Results ({chunks.length})</h3>
          {chunks.map((chunk, index) => (
            <ChunkCard key={index} chunk={chunk} />
          ))}
        </div>
      )}

      {graphContext && graphContext.queryPaths && (
        <div className="graph-context">
          <h3>Related Knowledge Graph Paths</h3>
          <pre>{JSON.stringify(graphContext.queryPaths, null, 2)}</pre>
        </div>
      )}
    </div>
  );
};

const ChunkCard = ({ chunk }) => {
  return (
    <div className="source-card">
      <div className="source-header">
        <span className="source-type">{chunk.source}</span>
        <span className="source-title">{chunk.source_title}</span>
        <span className="source-date">{formatDate(chunk.timestamp)}</span>
      </div>

      <div className="source-chunk">
        <p>{chunk.chunk_content}</p>
        {chunk.bounding_box && (
          <div className="chunk-highlight">
            Position: {chunk.bounding_box.x}, {chunk.bounding_box.y}
          </div>
        )}
      </div>
    </div>
  );
};
```

## Step 4: Data Synchronization

### 4.1 Scheduled Sync Jobs

```javascript theme={null}
class DataSyncManager {
  constructor(connectors, cortexIngestion) {
    this.connectors = connectors;
    this.cortexIngestion = cortexIngestion;
    this.syncIntervals = {
      slack: 5 * 60 * 1000, // 5 minutes
      gmail: 10 * 60 * 1000, // 10 minutes
      notion: 30 * 60 * 1000, // 30 minutes
      documents: 60 * 60 * 1000 // 1 hour
    };
  }

  async startSync() {
    // Start sync jobs for each connector
    Object.entries(this.syncIntervals).forEach(([connector, interval]) => {
      setInterval(() => {
        this.syncConnector(connector);
      }, interval);
    });
  }

  async syncConnector(connectorName) {
    try {
      const connector = this.connectors[connectorName];
      const newData = await connector.fetchNewData();
      
      if (newData.length > 0) {
        const normalizedData = newData.map(item => 
          this.normalizeData(item, connectorName)
        );
        
        // Use batch upload with verification
        await this.cortexIngestion.uploadWithVerification(normalizedData);
        console.log(`Synced ${newData.length} items from ${connectorName}`);
      }
    } catch (error) {
      console.error(`Sync failed for ${connectorName}:`, error);
      // Implement retry logic with exponential backoff
      await this.retrySync(connectorName, error);
    }
  }

  async retrySync(connectorName, error, attempt = 1) {
    const maxAttempts = 3;
    const delay = Math.pow(2, attempt) * 1000; // Exponential backoff
    
    if (attempt < maxAttempts) {
      console.log(`Retrying sync for ${connectorName} in ${delay}ms (attempt ${attempt + 1})`);
      setTimeout(() => {
        this.syncConnector(connectorName);
      }, delay);
    } else {
      console.error(`Max retry attempts reached for ${connectorName}:`, error);
    }
  }

  normalizeData(item, sourceType) {
    return {
      id: `${sourceType}_${item.id}`,
      title: item.title || item.subject || item.text?.substring(0, 100),
      source: sourceType,
      timestamp: item.timestamp || item.created_at || new Date().toISOString(),
      content: {
        text: item.text || item.body || item.content,
        html_base64: item.html ? btoa(item.html) : '',
        markdown: item.markdown || ''
      },
      url: item.url,
      description: item.description,
      metadata: {},
      additional_metadata: {
        id: item.id,
        author: item.author || item.user,
        tags: item.tags || [],
        created_at: item.created_at,
        updated_at: item.updated_at
      }
    };
  }
}
```

### 4.2 Webhook Integration

<CodeGroup>
  ```python Python SDK theme={null}
  import os
  import json
  from hydra_db import HydraDB
  from flask import Flask, request

  app = Flask(__name__)
  client = HydraDB(token=os.environ["HYDRA_DB_API_KEY"])

  @app.post("/webhooks/slack")
  def slack_webhook():
      event = request.json.get("event", {})

      if event.get("type") == "message":
          from datetime import datetime
          normalized_data = {
              "id": f"slack_{event['ts']}",
              "database": os.environ["TENANT_ID"],
              "collection": os.environ["SUB_TENANT_ID"],
              "title": f"Message in {event['channel']}",
              "type": "slack_message",
              "content": {"text": event["text"]},
              "metadata": {},
              "additional_metadata": {
                  "id": event["ts"],
                  "author": event["user"],
                  "created_at": datetime.utcfromtimestamp(float(event["ts"])).isoformat(),
                  "channel": event["channel"]
              }
          }

          client.context.ingest(
              database=os.environ["TENANT_ID"],
              app_knowledge=json.dumps([normalized_data])
          )

      return "OK", 200
  ```

  ```typescript TypeScript SDK theme={null}
  import { HydraDBClient } from "@hydradb/sdk";

  const client = new HydraDBClient({ token: process.env.HYDRA_DB_API_KEY! });

  // Express.js webhook handler
  app.post("/webhooks/slack", async (req, res) => {
    const { event } = req.body;

    if (event.type === "message") {
      const normalizedData = {
        id: `slack_${event.ts}`,
        database: process.env.TENANT_ID!,
        collection: process.env.SUB_TENANT_ID!,
        title: `Message in ${event.channel}`,
        type: "slack_message",
        content: { text: event.text },
        metadata: {},
        additional_metadata: {
          id: event.ts,
          author: event.user,
          created_at: new Date(event.ts * 1000).toISOString(),
          channel: event.channel
        }
      };

      await client.context.ingest({
        database: process.env.TENANT_ID!,
        appKnowledge: JSON.stringify([normalizedData])
      });
    }

    res.status(200).send("OK");
  });
  ```
</CodeGroup>

## Step 5: Security and Access Control

### 5.1 Multi-Tenant Architecture

```javascript theme={null}
class TenantManager {
  constructor() {
    this.tenants = new Map();
  }

  async createTenant(database, config) {
    const tenant = {
      id: database,
      subTenants: new Map(),
      permissions: config.permissions || {},
      dataSources: config.dataSources || []
    };
    
    this.tenants.set(database, tenant);
    return tenant;
  }

  async createSubTenant(database, collection, config) {
    const tenant = this.tenants.get(database);
    if (!tenant) throw new Error('Database not found');

    const subTenant = {
      id: collection,
      parentTenant: database,
      permissions: config.permissions || {},
      dataSources: config.dataSources || []
    };

    tenant.subTenants.set(collection, subTenant);
    return subTenant;
  }

  async searchWithTenantContext(query, database, collection = null, userId = null) {
    const tenant = this.tenants.get(database);
    if (!tenant) throw new Error('Database not found');

    // Check user permissions
    if (userId && !this.hasPermission(tenant, collection, userId)) {
      throw new Error('Access denied');
    }

    const searchOptions = {
      database,
      collection,
      metadata: {
        database: database
      }
    };

    if (collection) {
      searchOptions.metadata.collection = collection;
    }

    return await searchClient.search(query, searchOptions);
  }

  hasPermission(tenant, collection, userId) {
    // Implement your permission logic here
    return true; // Simplified for example
  }
}
```

### 5.2 Data Privacy and Compliance

```javascript theme={null}
class DataPrivacyManager {
  constructor() {
    this.retentionPolicies = new Map();
  }

  async applyRetentionPolicy(database, policy) {
    const { retentionDays, dataTypes, autoDelete } = policy;

    if (autoDelete) {
      const cutoff = new Date();
      cutoff.setDate(cutoff.getDate() - retentionDays);

      // DELETE /context deletes by explicit id - there is no date filter - so
      // list the memories first and delete only those older than the cutoff.
      const expiredIds = await this.findExpiredMemories(database, cutoff);
      if (expiredIds.length > 0) {
        await this.deleteOldData(database, expiredIds);
      }
    }
  }

  async findExpiredMemories(database, cutoff, collection = null) {
    const body = { type: 'memory', database: database };
    if (collection) {
      body.collection = collection;
    }

    const response = await fetch('https://api.hydradb.com/context/list', {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${API_KEY}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(body)
    });

    const envelope = await response.json();
    // A type: 'memory' listing returns data.user_memories, not data.sources, and
    // each row's identifier is memory_id (knowledge listings use data.sources
    // with id). Reading data.sources here would always yield an empty array.
    const memories = envelope.data?.user_memories ?? [];

    return memories
      .filter((memory) => memory.timestamp && new Date(memory.timestamp) < cutoff)
      .map((memory) => memory.memory_id);
  }

  async deleteOldData(database, ids, collection = null) {
    // Use HydraDB's context deletion endpoint for memory deletion.
    // Raw fetch to the REST API: the JSON body uses snake_case wire keys.
    const body = { type: 'memory', database: database, ids: ids };
    if (collection) {
      body.collection = collection;
    }

    const response = await fetch('https://api.hydradb.com/context', {
      method: 'DELETE',
      headers: {
        'Authorization': `Bearer ${API_KEY}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(body)
    });

    return response.json();
  }

  async anonymizeData(data, anonymizationRules) {
    // Implement data anonymization based on rules
    let anonymizedData = { ...data };
    
    anonymizationRules.forEach(rule => {
      if (rule.field && anonymizedData[rule.field]) {
        anonymizedData[rule.field] = this.anonymizeValue(
          anonymizedData[rule.field], 
          rule.method
        );
      }
    });
    
    return anonymizedData;
  }

  anonymizeValue(value, method) {
    switch (method) {
      case 'hash':
        return crypto.createHash('sha256').update(value).digest('hex');
      case 'mask':
        return value.replace(/./g, '*');
      case 'redact':
        return '[REDACTED]';
      default:
        return value;
    }
  }
}
```

## Step 6: Performance Optimization

### 6.1 Caching Strategy

```javascript theme={null}
class SearchCache {
  constructor() {
    this.cache = new Map();
    this.ttl = 5 * 60 * 1000; // 5 minutes
  }

  async getCachedResults(query, filters) {
    const key = this.generateCacheKey(query, filters);
    const cached = this.cache.get(key);
    
    if (cached && Date.now() - cached.timestamp < this.ttl) {
      return cached.results;
    }
    
    return null;
  }

  async setCachedResults(query, filters, results) {
    const key = this.generateCacheKey(query, filters);
    this.cache.set(key, {
      results,
      timestamp: Date.now()
    });
  }

  generateCacheKey(query, filters) {
    return `${query}_${JSON.stringify(filters)}`;
  }

  clearExpired() {
    const now = Date.now();
    for (const [key, value] of this.cache.entries()) {
      if (now - value.timestamp > this.ttl) {
        this.cache.delete(key);
      }
    }
  }
}
```

### 6.2 Rate Limiting

```javascript theme={null}
class RateLimiter {
  constructor(limit, windowMs) {
    this.limit = limit;
    this.windowMs = windowMs;
    this.requests = new Map();
  }

  async checkLimit(userId) {
    const now = Date.now();
    const userRequests = this.requests.get(userId) || [];
    
    // Remove old requests outside the window
    const validRequests = userRequests.filter(
      timestamp => now - timestamp < this.windowMs
    );
    
    if (validRequests.length >= this.limit) {
      return false; // Rate limit exceeded
    }
    
    // Add current request
    validRequests.push(now);
    this.requests.set(userId, validRequests);
    
    return true; // Request allowed
  }
}
```

## Step 7: Monitoring and Analytics

### 7.1 Search Analytics

```javascript theme={null}
class SearchAnalytics {
  constructor() {
    this.metrics = {
      searches: 0,
      successfulSearches: 0,
      failedSearches: 0,
      averageResponseTime: 0,
      popularQueries: new Map(),
      sourceTypeUsage: new Map()
    };
  }

  async trackSearch(query, filters, results, responseTime) {
    this.metrics.searches++;
    
    if (results && results.answer) {
      this.metrics.successfulSearches++;
    } else {
      this.metrics.failedSearches++;
    }

    // Track popular queries
    const queryKey = query.toLowerCase().trim();
    this.metrics.popularQueries.set(
      queryKey, 
      (this.metrics.popularQueries.get(queryKey) || 0) + 1
    );

    // Track source type usage
    if (results && results.data?.chunks) {
      results.data.chunks.forEach(chunk => {
        const sourceType = chunk.source;
        this.metrics.sourceTypeUsage.set(
          sourceType,
          (this.metrics.sourceTypeUsage.get(sourceType) || 0) + 1
        );
      });
    }

    // Update average response time
    this.updateAverageResponseTime(responseTime);
  }

  updateAverageResponseTime(newTime) {
    const currentAvg = this.metrics.averageResponseTime;
    const totalSearches = this.metrics.searches;
    
    this.metrics.averageResponseTime = 
      (currentAvg * (totalSearches - 1) + newTime) / totalSearches;
  }

  getMetrics() {
    return {
      ...this.metrics,
      popularQueries: Array.from(this.metrics.popularQueries.entries())
        .sort((a, b) => b[1] - a[1])
        .slice(0, 10),
      sourceTypeUsage: Array.from(this.metrics.sourceTypeUsage.entries())
        .sort((a, b) => b[1] - a[1])
    };
  }
}
```

## Best Practices and Recommendations

### 1. Data Ingestion Best Practices

* **Batch Processing**: Use HydraDB's batch upload endpoints for efficiency
* **Batch Limits**: Limit to 20 app sources per request with 1-second intervals between batches
* **Incremental Sync**: Only sync new/changed data to minimize API calls
* **Error Handling**: Implement retry logic with exponential backoff
* **Processing Verification**: Always verify upload processing using `/context/status`
* **Rate Limiting**: Respect API rate limits and implement queuing

### 2. Search Optimization

* **Query Preprocessing**: Clean and normalize user queries
* **Result Ranking**: Use `alpha` and `recency_bias` for fine-tuning
* **Metadata Filtering**: Use `source_title` and `source_type` for targeted searches
* **Thinking Mode**: Use `mode: "thinking"` for complex queries that benefit from multi-query retrieval with reranking
* **Caching**: Cache frequent queries and results

### 3. Security Considerations

* **Data Encryption**: Encrypt sensitive data at rest and in transit
* **Access Control**: Implement role-based access control (RBAC)
* **Audit Logging**: Log all search queries and data access
* **Data Retention**: Implement automatic data deletion policies

### 4. Performance Tips

* **Connection Pooling**: Reuse HTTP connections
* **Async Processing**: Use async/await for non-blocking operations
* **Memory Management**: Implement proper cleanup for large datasets
* **Batch Optimization**: Respect 20-source batch limits and 1-second intervals
* **Processing Verification**: Verify uploads to ensure data is properly indexed
* **Monitoring**: Track response times and error rates

### 5. User Experience

* **Autocomplete**: Implement search suggestions
* **Faceted Search**: Allow filtering by source type, date, author
* **Saved Searches**: Let users save and share search queries
* **Export Results**: Allow users to export search results

## Deployment Checklist

* Set up authentication and authorization
* Configure data connectors and sync schedules
* Implement error handling and monitoring
* Set up caching and rate limiting
* Configure backup and disaster recovery
* Test with production data volumes
* Set up analytics and reporting
* Document API usage and troubleshooting

## Conclusion

Building a Glean-like application with HydraDB APIs provides you with a powerful, scalable foundation for workplace search and AI assistance. By following this guide and implementing the best practices outlined, you can create a comprehensive solution that rivals commercial offerings while maintaining full control over your data and user experience.

The key to success is starting with a solid architecture, implementing proper data synchronization, and gradually adding advanced features like multi-step reasoning, conversation memory, and personalized responses. HydraDB's APIs provide the AI capabilities you need, while your application handles the data ingestion, user interface, and business logic.

Remember to monitor performance, gather user feedback, and continuously iterate on your implementation to create the best possible user experience.

## Change Log

| Date       | Change                                                                                                                                  |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-05-14 | Replaced raw `fetch()`-based `HydraDBDataIngestion` class with official SDK calls (`client.context.ingest`, `client.context.status`)    |
| 2026-05-14 | Replaced raw `fetch()`-based `GleanSearch` class with `client.query` SDK call                                                           |
| 2026-05-14 | Replaced `AdvancedSearch` class with standalone SDK helper functions (`searchWithFilters`, `searchWithContext`, `conversationalSearch`) |
| 2026-05-14 | Replaced `PersonalizedSearch` class with flat SDK-based `searchWithMemory` function                                                     |
| 2026-05-14 | Replaced webhook `cortexIngestion.uploadBatch()` call with `client.context.ingest()` SDK call                                           |
| 2026-05-14 | Added Python SDK equivalents in `<CodeGroup>` tabs for all replaced code blocks                                                         |
| 2026-05-14 | Removed "Status: In progress" notice                                                                                                    |
