Lucid Sensor

What it ingests

Lucidchart diagrams and Lucidspark boards, through the Lucid REST API (https://api.lucid.co):

Lucid object	Ontology entity	Signal
Lucidchart document	CreativeWork (source_type: diagram)	document
Lucidspark board	CreativeWork (source_type: whiteboard)	document
Lucidscale model (opt-in)	CreativeWork (source_type: architecture_model)	document
Creator, last editor	Person (identity)	—

And these edges:

• the last editor AUTHORED the document;
• the creator also AUTHORED it, when that is someone other than the last editor.

People resolve to existing Person vertices by email, so a Lucid editor who is also a Slack author or a GitHub committer is one person in the graph. Emails come from the account's user directory; see People for what happens when the credential cannot read it.

Each document's text is its title plus the text on its shapes and connectors, page by page. A labelled connector is written as Cart service → Payments API: charge, so the relationships a diagram draws survive as prose the graph can search. Text is capped at 60,000 characters per document, and secret-shaped strings (API keys, tokens) are redacted before it is stored.

Document properties include the title, link, product, version, page count, created and last-modified times, status, classification, custom tags and parent folder id.

External ids are stable: lucid:document:<document id>. An edit updates the same entity rather than creating another. A document's name is its title, except that a Lucid default title (Untitled Document, Blank diagram, …) gets the document id appended so it does not merge with every other untitled document in the organization. A renamed document keeps the name it was first ingested under, so its edges keep pointing at the same entity; its current title is the title property.

The mapping is deterministic. No LLM runs per document.

Credential

References a lucid_credentials credential by id. The secret is a JSON object in one of two forms. Both read the documents that one Lucid user can see: their own documents, and whatever has been shared with them directly, through a team folder or through a repository.

An API key (simplest):

{ "api_key": "…" }

An OAuth 2.0 user token, when your Lucid admins prefer an app they can review and revoke:

{
  "client_id": "…",
  "client_secret": "…",
  "refresh_token": "oauth2-…"
}

Lucid refresh tokens are single-use: every refresh returns a new one and invalidates the old one. Corveil writes the new token back to the same vault row after each refresh, so the stored secret changes over time. Don't reuse the same refresh token in another tool, or point two Lucid sensors at one OAuth credential: whichever refreshes second is locked out. An API key has no such limit.

curl -sS -X POST https://your-corveil-host/api/credentials \
  -H "Authorization: Bearer <admin>" -H "Content-Type: application/json" \
  -d '{"type":"lucid_credentials","name":"acme-lucid","secret":"{\"api_key\":\"…\"}"}'

The validator rejects a bare key that isn't wrapped in JSON, both forms in one object, a partial OAuth client, and a pasted Bearer … header.

Permissions

What	API key grant	OAuth 2.0 user token scope
Search and read Lucidchart documents	DocumentReadonly	lucidchart.document.content:readonly
Search and read Lucidspark boards	DocumentReadonly	lucidspark.document.content:readonly
Lucidscale models (only if you add lucidscale)	DocumentReadonly	lucidscale.document.content:readonly
Resolve creators and editors to emails (optional)	the account user-read grant (an account owner or admin must grant it)	account.users:admin.readonly (an account owner or admin must grant it)
Refresh without the user signing in again	—	offline_access

Everything is read-only. The sensor never creates, edits, shares or trashes anything.

People

Lucid reports a document's creator and last editor by numeric user id. To turn those ids into names and emails, the sensor reads the account's user directory (GET /users) once per poll. That needs the admin-granted permission in the table above.

Without it, the poll still succeeds. The sensor logs user directory unavailable and falls back to the document owner's name, which search returns on its own. A creator who owns the document is linked by name only, and anyone else gets no AUTHORED edge. A name-only person is not merged into an existing Person by email.

Config fields

Key	Required	Notes
products	no	Comma-separated: lucidchart, lucidspark, lucidscale. Blank means Lucidchart and Lucidspark. Unknown names are rejected at configure time.
include_content	no	true (default) fetches each changed document's shape and connector text. false ingests titles and metadata only, skipping the per-document contents request.
lookback_days	no	How far back the first sync reaches by last-modified date. Default 90, maximum 3650.
max_documents	no	Most documents ingested per poll. Default 200, maximum 2000. A larger backlog resumes on the next poll.

There is no default_entity_type. Documents always land as CreativeWork.

Provider-side setup

Both routes need Lucid's developer tools, which an account admin unlocks for the user; see Lucid's guide. Use a dedicated service user if you can, and share the folders you want ingested with it. The sensor sees exactly what that user sees.

Option A — API key

1. Signed in as that user, open the Lucid Developer Portal.
2. Create an API key. Give it the DocumentReadonly grant. To resolve people by email, have an account admin add the account user-read grant too.
3. Copy the key and store it as {"api_key":"…"}.

Option B — OAuth 2.0 user token

1. In the Developer Portal, create an application, then add an OAuth 2.0 client to it. Copy the client ID and client secret.

2. Register a redirect URI. Lucid's test redirect URI, https://lucid.app/oauth2/clients/<client_id>/redirect, shows the code on screen, so you don't need a server for a one-time grant.

3. An unpublished app only works for its collaborators, so add the user who will authorize it as an app collaborator.

4. Signed in as that user, open this URL in a browser (one line), approve it, and copy the code from the page it redirects to:

   https://lucid.app/oauth2/authorize?client_id=<client_id>
     &redirect_uri=https://lucid.app/oauth2/clients/<client_id>/redirect
     &scope=lucidchart.document.content:readonly%20lucidspark.document.content:readonly%20account.users:admin.readonly%20offline_access

   Leave out account.users:admin.readonly if an admin won't grant it, and add lucidscale.document.content:readonly if you set products to include Lucidscale.

5. Exchange the code within five minutes:

   curl -sS https://api.lucid.co/oauth2/token -X POST \
     -H "Content-Type: application/json" \
     -d '{"grant_type":"authorization_code","code":"<code>","client_id":"<client_id>","client_secret":"<client_secret>","redirect_uri":"https://lucid.app/oauth2/clients/<client_id>/redirect"}'

6. Store client_id, client_secret and the response's refresh_token as the credential. Ignore the access_token: it lasts an hour, and the sensor mints its own.

Use the user-token authorize URL (/oauth2/authorize), not the account-token one (/oauth2/authorizeAccount). Lucid's document search only accepts user tokens and API keys.

Wire-up

POST /api/sensors

{
  "sensor_type": "lucid",
  "name": "Acme Lucid",
  "config": {
    "products": "lucidchart, lucidspark",
    "include_content": "true"
  },
  "credentials": { "credentials": "<lucid_credentials credential id>" }
}

Verify

Run Test connection. It runs a one-page document search for the configured products, so a missing grant or scope fails at save/test (as an auth error) rather than on the first poll.

Once polling, each poll logs lucid sensor: poll complete with documents_collected, documents_unchanged, documents_without_text and terminal_reason. In the graph, search for a diagram's title: it should be a CreativeWork with source_type: diagram, AUTHORED by its last editor.

How incremental sync works

• Search by last-modified time. Each poll runs POST /documents/search with lastModifiedAfter set to the watermark, minus a 10-minute overlap that covers clock skew and search-index lag. Trashed documents are excluded.
• Unchanged documents in the overlap are skipped. The cursor remembers the version of each document it ingested inside the overlap, so the overlap costs a search page, not another contents download.
• A backlog resumes across polls. Lucid orders search results by creation time, not modification time, so the sensor walks the results with Lucid's page token. When max_documents (or 40 pages) is reached, the token is saved and the next poll resumes there. The watermark only moves once the walk reaches the last page, and then to the time the walk started. A document edited mid-walk on a page already passed is picked up by the next search.
• A new version is a new event, not a new document. Event ids carry Lucid's document version (lucid:document:<id>@v<version>), so an edit is ingested. The entity id stays lucid:document:<id>, so the edit updates the entity it already has.
• A document the user lost access to (Lucid answers 403 on its contents) is still ingested with its metadata, without text.
• An expired page token restarts the walk from the watermark instead of wedging the sensor.
• Rate limits. Lucid allows 15 requests a second per user token and 100 contents requests per 5 seconds per account. The sensor paces itself at 10 requests a second. On a 429 it waits out a Retry-After of up to 30 seconds, up to three times. A longer limit ends the poll, keeps what it already fetched, and leaves the cursor where it stopped.
• Sync cadence is the sensor's poll interval. Edits appear within one interval of Lucid's search index catching up.

Known limits

• One user's view, not the whole account. Lucid's account-wide document search is exclusive to Enterprise Shield and uses admin scopes. The sensor doesn't use it. Share the folders you want ingested with the sensor's user.
• Deletes are not propagated. A trashed or deleted document stays in the graph. The ontology has no delete path for sensor events.
• Collaborators and comments are not ingested. Only the creator and the last editor become people.
• Folders are recorded by id only. A document's folder_id is a property; folders are not entities.
• Images, embedded data and linked data sets aren't read. Only text on shapes and connectors becomes content.
• Lucid's FedRAMP environment is not supported. The sensor only talks to api.lucid.co.
• A rotated refresh token that never reaches the vault (for example, a database outage at the moment of refresh) leaves the stored token spent. The next poll fails as an auth error; re-run the OAuth grant (Option B, steps 4–6). An unused refresh token also expires after 180 days.

Not included

• Webhooks. Polling search covers every document the user can see without a public endpoint.
• Document exports (PNG / PDF) and Lucid's embedding APIs.
• Writing back to Lucid (Mermaid diagrams, comments, sharing).