S3 Bucket Source

What it ingests

Objects from a bucket you own, optionally narrowed to a key prefix. Each object becomes one CreativeWork in the ontology, carrying its key, size, content type and modification time.

This is the source for the files your systems drop rather than the documents your people author: nightly exports, generated reports, meeting transcripts, knowledge dumps, data-room contents. For many organizations that bucket is where the source of truth actually lives, and until now Corveil had no path to it.

Body text is extracted from text-like formats — .md, .txt, .csv, .tsv, .json, .yaml, .xml, .html, .log, .sql, .vtt, .srt — plus .pdf and .docx. Everything else still lands as an entity with its key, URI, size and metadata, just without a body.

This is not Corveil's own storage

Corveil uses S3 internally for worker-run artifacts, chat attachments and org exports. Those buckets are Corveil infrastructure and are configured by your operator, not through this page. This source is a connector to a bucket you own.

Credential

References an aws_access_key credential — a JSON object holding an IAM access key pair:

{
  "access_key_id": "AKIAIOSFODNN7EXAMPLE",
  "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

session_token is a third, optional field, required only for temporary (STS-issued) credentials.

Prefer no credential at all

The credential field is optional, and leaving it blank is the better choice whenever Corveil runs in AWS. Blank falls back to the deployment's own identity — the ECS task role, the EKS service account (IRSA), the EC2 instance role, or the environment — so there is no long-lived secret to store, leak or rotate. Grant the policy below to that role instead.

Store an access key when Corveil cannot assume an identity that reaches the bucket: a bucket in a different AWS account with no cross-account role, or an S3-compatible store that is not AWS at all.

Temporary credentials expire

An access key ID beginning ASIA is an STS credential. It needs its session_token, and it stops working within hours — which stops the source. Corveil refuses to store one without a token, but it cannot stop one from expiring. Use a long-term (AKIA) key, or better, the ambient-credential path above.

Config fields

Key	Required	Notes
bucket	yes	The bucket name alone — not an s3:// URI, not an ARN.
prefix	no	Only objects whose key starts with this string are listed. Blank ingests the whole bucket.
region	no	The bucket's region. Blank uses the region from the deployment's AWS environment.
endpoint	no	For S3-compatible stores. Blank targets Amazon S3.
recursive	no	true (default) descends the whole key path under the prefix; false ingests only the objects directly under it.
path_style	no	Overrides the default, which is on when endpoint is set and off for Amazon S3.

About prefix

prefix is a literal string prefix, not a folder. reports matches reports/q1.pdf and reports-archive.zip. End it with / when you mean a folder:

prefix: "reports/"

A leading / is stripped, because S3 keys do not begin with one and a prefix of /reports/ would otherwise match nothing at all — a source that looks healthy and ingests forever nothing.

The prefix is applied by S3 itself, not filtered afterwards, so objects outside it are never listed, never read, and never billed for.

Provider-side setup

The IAM policy

Two actions, scoped to the one prefix. Nothing else is needed, and nothing else should be granted:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyThePrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-knowledge-exports",
      "Condition": {
        "StringLike": { "s3:prefix": ["reports/*", "reports/"] }
      }
    },
    {
      "Sid": "ReadObjectsUnderThePrefix",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::acme-knowledge-exports/reports/*"
    }
  ]
}

Two details are easy to get wrong:

• s3:ListBucket targets the bucket ARN; s3:GetObject targets the object ARN. They differ by the /*. A policy that puts both on the same resource grants neither.
• The s3:prefix condition needs both patterns. reports/* alone rejects the request Corveil makes for the prefix itself.

To ingest the whole bucket, drop the Condition block and use arn:aws:s3:::acme-knowledge-exports/* as the GetObject resource.

For GovCloud, the partition changes: arn:aws-us-gov:s3:::….

A list-only policy still works

Grant s3:ListBucket without s3:GetObject and the source ingests the bucket's inventory — every object's key, size, type and modification time — with no bodies. That is a legitimate configuration for a bucket whose contents are too sensitive to read, and it fails softly: bodiless documents, not a broken source.

KMS-encrypted buckets

Objects encrypted with SSE-S3 need nothing extra. Objects encrypted with a customer-managed KMS key also need kms:Decrypt on that key, granted to the same principal:

{
  "Effect": "Allow",
  "Action": "kms:Decrypt",
  "Resource": "arn:aws:kms:us-gov-west-1:111122223333:key/<key-id>"
}

Without it, listing still succeeds and every body fetch fails — so the symptom is an inventory of bodiless documents, the same shape as a missing s3:GetObject.

S3-compatible stores

Set endpoint to the store's URL and the rest is unchanged:

Store	endpoint	Notes
MinIO	https://minio.internal:9000	Uses path-style addressing, which endpoint enables automatically.
Ceph RADOS Gateway	https://rgw.internal	As above.
Cloudflare R2	https://<account>.r2.cloudflarestorage.com	Set region to auto.

Setting endpoint switches to path-style addressing (host/bucket/key) because self-hosted gateways generally cannot serve the virtual-host form (bucket.host/key) without wildcard DNS and a wildcard certificate. If your store does support virtual-host addressing, set path_style to false.

Region is often meaningless to these stores; when you leave it blank alongside an endpoint, Corveil signs with us-east-1, which is what MinIO and most others expect.

Wire-up

POST /api/sensors

A prefix in a bucket, using the deployment's own AWS identity:

{
  "sensor_type": "s3",
  "name": "Knowledge exports",
  "config": {
    "bucket": "acme-knowledge-exports",
    "prefix": "reports/",
    "region": "us-gov-west-1"
  },
  "credentials": {}
}

A cross-account bucket with a stored access key:

{
  "sensor_type": "s3",
  "name": "Partner data room",
  "config": {
    "bucket": "partner-dataroom",
    "prefix": "shared/corveil/",
    "region": "us-east-1"
  },
  "credentials": { "aws_credential": "<aws_access_key credential id>" }
}

MinIO, top level only:

{
  "sensor_type": "s3",
  "name": "MinIO drop",
  "config": {
    "bucket": "drop",
    "endpoint": "https://minio.internal:9000",
    "recursive": false
  },
  "credentials": { "aws_credential": "<aws_access_key credential id>" }
}

Verify

Run Test connection. It performs a single-key ListObjectsV2 against the configured bucket and prefix — the same call the poll loop makes — so a policy that grants bucket-wide listing but not your prefix fails here rather than at the first poll.

Credential failures name the AWS error rather than a generic one: InvalidAccessKeyId, SignatureDoesNotMatch, ExpiredToken, AccessDenied, NoSuchBucket, and PermanentRedirect (the bucket is in a different region than region says) each come back with the code and the remedy.

A reachable but empty prefix reports as healthy, and says so explicitly — a drop bucket legitimately sits empty between deliveries, and that must not read the same as a mistyped prefix.

Sources only begin polling once you confirm their bindings.

Notes and limits

The first sweep drains across several polls. S3 has no server-side "changed since" filter — the only server-side narrowing is the key prefix, and results come back in key order, never in modification order. So Corveil re-walks the prefix each poll and compares modification times itself. Each poll is bounded; when a prefix is larger than one poll's budget, the source records the key it reached and resumes there next tick. A large bucket therefore fills in progressively rather than arriving as one enormous poll or restarting from the top each time.

Modified objects are re-ingested; the document is not duplicated. Overwriting an object produces a new event (keyed by its ETag) that refreshes the existing CreativeWork rather than creating a second one. A report overwritten nightly is one document in the graph, not 365.

Deleted objects are not removed from the ontology. A deleted object simply stops appearing in the listing; entities already created from it remain.

Objects have no author. Unlike Google Drive or SharePoint, a listed S3 object carries no usable identity — its only owner field is an opaque AWS canonical user ID, not a name or an email. Documents from this source are therefore unattributed, and no Person is created. Identity, where it matters, comes from the same document arriving through a source that knows who wrote it.

Objects with no file extension are ingested without a body. There is no way to know a key's type without fetching it, and fetching every extensionless key to discover it is a video would be worse. Give exported files an extension if you want their text.

.xlsx and .pptx land without a body. Like .docx they are ZIP containers, but Corveil's extractor understands only Word's document XML today. They still ingest with key, size and metadata.

Objects over 10 MB are ingested without a body, matching the Google Drive source's export ceiling.

Archived objects are not read. GLACIER and DEEP_ARCHIVE objects need a restore before they can be fetched, so Corveil lists them — you will see them in the ontology, with their storage class — but does not attempt a download.

Bucket versioning is not traversed. The source reads the current version of each key. Previous versions are not enumerated.

Event-driven ingestion (S3 Event Notifications → SQS) is not implemented. Polling was chosen first because it needs s3:ListBucket and nothing else, and it re-discovers anything that landed while Corveil was down. An event path would lower latency at the cost of a queue, a bucket notification config and a cross-account policy on your side.