Collection Profile

Date: 2026-08-14

Scope: How data is collected into a PDPP resource server: connector runtime, JSONL protocol, bindings, and sync state.

Companion to the Personal Data Portability Protocol (PDPP) core spec.

Overview

The Collection Profile defines how connectors collect data from source platforms and write it to a PDPP resource server. It is one fulfillment mechanism for the PDPP core protocol; pre-collected data, manual imports, and other ingestion mechanisms are equally valid.

The Collection Profile is architecturally separate from the core protocol. A resource server serving pre-collected data needs no awareness of this profile. A connector runtime implementing this profile needs no awareness of grant semantics beyond what is explicitly passed to it in the START message.

Collection method abstraction

Connectors abstract over the source platform's data access interface. The runtime does not standardize the connector's source-specific collection logic; it standardizes only the runtime contract around bindings, scope, state, and emitted messages. A connector that collects data via browser automation and one that calls a platform's export API both use the same START/RECORD/STATE/DONE protocol, the same binding matching, and the same state management.

This abstraction is intentional. Many platforms do not currently offer structured data portability APIs. The browser_automation binding enables connectors that drive a browser to collect data from a platform's web UI. As platforms adopt data portability standards or offer their own APIs, connector implementations can change without changing the consent surface, grant enforcement, or query API.

Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this profile are to be interpreted as described in the core spec's Requirements Language (BCP 14 [RFC 2119] [RFC 8174]) when, and only when, they appear in all capitals.

1. Source Declaration Extension

The Core SourceDeclaration (Section 5 of the core spec) defines the common consent, record, selection, and query surface. This profile owns the https://pdpp.dev/profile/collection extension value, which adds connector acquisition and execution fields. Core preserves or ignores this value without interpreting it.

{
  "protocol_version": "0.1.0",
  "source": {
    "kind": "connector",
    "id": "https://registry.pdpp.dev/connectors/spotify"
  },
  "declaration_version": "2026-08-11",
  "publisher": {
    "id": "https://registry.pdpp.dev/"
  },
  "display": { "name": "Spotify" },
  "streams": [
    {
      "name": "top_artists",
      "semantics": "mutable_state",
      "schema": {
        "type": "object",
        "properties": { "id": { "type": "string" } },
        "required": ["id"]
      },
      "primary_key": ["id"],
      "selection": { "fields": true, "resources": false }
    }
  ],
  "extensions": {
    "https://pdpp.dev/profile/collection": {
      "connector": {
        "id": "https://registry.pdpp.dev/connectors/spotify-browser",
        "version": "2.0.0"
      },
      "runtime_requirements": {
        "bindings": {
          "network": { "required": true },
          "interactive": { "required": true }
        }
      },
      "capabilities": {
        "human_interaction": ["credentials", "otp"]
      },
      "streams": [
        { "name": "top_artists", "incremental": true }
      ]
    }
  }
}

Collection extension fields

The Collection extension value is a closed object. It MUST contain exactly the required connector and runtime_requirements members and MAY also contain capabilities and streams. Unknown top-level members are rejected.

• connector is closed and MUST contain exactly non-empty id and version. id MUST be an absolute URI.
• runtime_requirements is closed and MUST contain exactly bindings. bindings is an object keyed by a standard or namespaced binding identifier. A connector with no binding requirements MUST use "bindings": {}; it MUST NOT omit runtime_requirements or bindings. Each binding requirement MUST contain required, a boolean. A standard binding requirement MAY contain only the descriptor members registered for that binding; a namespaced binding owns its additional descriptor members.
• capabilities, when present, is closed and MUST contain exactly human_interaction, a non-empty unique array containing only credentials, otp, or manual_action.
• streams, when present, is a non-empty array of closed objects. Each entry MUST contain exactly name and incremental, where name is non-empty and unique in the extension and incremental is boolean.

The extension's streams[].name values MUST each match a stream in the enclosing SourceDeclaration. Collection fields MUST NOT redefine the Core schema, selection, consent, or query meaning of that stream. A provider_native SourceDeclaration MUST NOT use this extension. A connector SourceDeclaration remains valid Core input when the extension is absent.

Standard bindings

Every declaration requirement has required: boolean. The table lists the additional descriptor members registered for that binding. Runtime-assigned members such as ws_url and profile_path are normally omitted from the declaration and populated by the runtime in START.bindings. A START binding descriptor omits the declaration-only required member.

2. Connector Run Protocol

Connectors communicate with the runtime via newline-delimited JSON (JSONL) over stdin/stdout. Each message is a single JSON object followed by a newline.

Runtime binding matching

Before spawning a connector, the runtime checks the Collection extension's runtime_requirements.bindings against its own capabilities. If the runtime cannot satisfy a required binding, the run MUST fail with a clear error before the connector process is spawned. This follows the Kubernetes scheduler pattern: connectors declare requirements, runtimes advertise capabilities.

Connector process state machine

The connector process transitions through the following states:

State transition table:

Protocol violations:

• A connector MUST NOT emit INTERACTION while already in waiting_for_interaction. A runtime that receives a second INTERACTION in this state MUST terminate the connector process and mark the run as failed. Note (non-normative): Runtimes that process connector messages sequentially via a single-threaded message queue may make this violation unrepresentable in practice, because the queue serializes INTERACTION processing. The protocol rule remains valid for correct connector behavior and for runtime architectures that dispatch messages concurrently.
• A connector that receives INTERACTION_RESPONSE while in collecting (no pending INTERACTION) SHOULD treat it as a fatal protocol error, write a diagnostic to stderr, and exit with non-zero status.
• START is exactly-once. It MUST be the first message sent by the runtime. A connector that receives START while in any state other than initializing MUST treat it as a fatal protocol error.

Runtime behavior on failure: The runtime MUST NOT persist STATE checkpoints from a run that terminates in the failed state. State is only persisted after a successful DONE.

SKIP_RESULT is a message emitted while in the collecting state. It does not cause a state transition.

3. Messages

Runtime to Connector

START

Initializes a collection run.

{
  "type": "START",
  "run_id": "run_abc123",
  "collection_mode": "incremental",
  "scope": {
    "streams": [
      {
        "name": "top_artists",
        "time_range": {
          "since": "2025-10-11T00:00:00Z"
        },
        "fields": [
          "id",
          "name",
          "genres",
          "popularity",
          "source_updated_at"
        ]
      }
    ]
  },
  "state": {
    "top_artists": { "last_updated": "2026-03-01T00:00:00Z" }
  },
  "bindings": {
    "browser_automation": {
      "interface": "cdp",
      "ws_url": "ws://127.0.0.1:39011/devtools/browser/abc"
    },
    "network": {}
  }
}

The START message does not include the raw grant or access token. It carries a normalized scope object instead. scope is not itself a grant and has no authorization force; it is the collection target for this run. For grant-driven runs, the runtime MUST derive scope from the grant, MUST NOT construct a scope broader than the grant permits, and MAY narrow it further according to local fulfillment policy (for example, collecting only the stale streams needed to satisfy the current request). For proactive runs, the runtime derives scope from user preferences or local policy.

scope fields

CollectionStream fields

START.scope carries normalized collection targets only. It does not include issuance-time concepts such as necessity or unresolved view names; the runtime resolves those before spawning the connector.

Connector obligations for scope:

• A connector MUST NOT emit RECORD messages for streams absent from scope.streams.
• If resources or time_range is present for a stream, the connector MUST apply those constraints before emitting RECORD messages for that stream.
• If fields is present for a stream, the connector MUST NOT emit additional top-level fields in RECORD data for that stream, except that it MAY include schema-required or ingest-required top-level fields if the runtime omitted them accidentally.
• A connector that cannot honor a declared resources, time_range, or fields constraint for a stream MUST either emit SKIP_RESULT with reason: "scope_not_supported" and omit records for the skipped target, or fail the run. It MUST NOT silently broaden or ignore the constraint.
• A connector MAY retrieve broader source-side data transiently when the source platform cannot filter precisely, but it MUST still emit RECORD messages consistent with scope.

Connector compliance is not the only enforcement backstop. The runtime and downstream write path MUST reject or discard emissions that fall outside the declared scope.

State management: State is maintained at two levels:

• Global state: Used and advanced only by proactive runs (no grant). Represents archival completeness for the user's data store.
• Grant-scoped state: Used and advanced by continuous grant runs, keyed by grant_id. The runtime reads and writes this namespace through GET/PUT /v1/state/{connector_id}?grant_id={grant_id}. It ensures recurring app syncs are incremental without interfering with global archival cursors.
• Single-use runs: Receive state: null. STATE messages emitted during single-use runs are not persisted.

bindings contains a descriptor for every binding declared required: true in the Collection extension. For every required binding, the runtime MUST include a valid descriptor. Connectors MUST treat a missing required binding as a fatal protocol error. Connectors MUST ignore unknown binding keys.

INTERACTION_RESPONSE

Reply to an INTERACTION request.

{
  "type": "INTERACTION_RESPONSE",
  "request_id": "req_001",
  "status": "success",
  "data": { "email": "user@example.com", "password": "..." }
}

On timeout, the runtime MUST send a response with status: "timeout" rather than leaving the connector blocked indefinitely.

Connector to Runtime

RECORD

A single data record. Same envelope as the core spec (Section 4).

{
  "type": "RECORD",
  "stream": "top_artists",
  "key": "4Z8W4fKeB5",
  "data": {
    "id": "4Z8W4fKeB5",
    "name": "Radiohead",
    "genres": ["alternative rock"],
    "popularity": 82,
    "source_updated_at": "2026-03-28T00:00:00Z"
  },
  "emitted_at": "2026-04-06T15:01:00Z"
}

The op field (upsert or delete) is a directive to the resource server and is not stored as part of the record data.

STATE

Checkpoint for incremental sync.

{
  "type": "STATE",
  "stream": "top_artists",
  "cursor": { "last_updated": "2026-03-28T00:00:00Z" }
}

The runtime persists STATE only after preceding records are durably written to the resource server. Connectors SHOULD emit STATE periodically (e.g., every 1000 records) rather than only at the end of a stream.

The cursor object is opaque to the runtime and the resource server: its structure is defined by the connector and interpreted only by the connector on the next run.

INTERACTION

Request input from a user or agent. The connector blocks (does not emit further messages) until INTERACTION_RESPONSE arrives on stdin.

{
  "type": "INTERACTION",
  "request_id": "req_001",
  "kind": "credentials",
  "message": "Log in to Spotify",
  "schema": {
    "type": "object",
    "properties": {
      "email": { "type": "string" },
      "password": { "type": "string", "format": "password" }
    },
    "required": ["email", "password"]
  },
  "timeout_seconds": 300
}

SKIP_RESULT

Signals that a stream or resource was intentionally skipped. Does not cause a state transition.

{
  "type": "SKIP_RESULT",
  "stream": "playlists",
  "reason": "rate_limited",
  "message": "Skipped playlists: rate limit reached"
}

SKIP_RESULT MAY also be used when a connector cannot honor a declared scope element for a stream or resource. In that case the reason MUST be scope_not_supported.

PROGRESS

Optional progress update for display in runtime UIs.

{
  "type": "PROGRESS",
  "stream": "messages",
  "message": "Downloaded 500 of 2196 messages",
  "count": 500,
  "total": 2196
}

DONE

Signals completion. Must be the final message emitted by the connector.

{
  "type": "DONE",
  "status": "succeeded",
  "records_emitted": 2196
}

On failure:

{
  "type": "DONE",
  "status": "failed",
  "records_emitted": 0,
  "error": { "message": "Authentication failed", "retryable": true }
}

4. Resource Server Transport

A Resource Server that claims Collection Profile support implements the owner-authenticated ingest and state endpoints in this section. These endpoints are not part of Core Resource Server conformance.

Ingest records

POST /v1/ingest/{stream}
Authorization: Bearer <owner_token>
Content-Type: application/x-ndjson

The body is NDJSON with one Core RECORD envelope per line.

The Resource Server MUST reject a record whose consent_time_field value is null, absent, or not a valid ISO 8601 datetime with 400 invalid_record. It MUST reject a record with 400 invalid_record_identity when the values of the data fields named by primary_key disagree with the corresponding values in the RECORD key. If legacy records contain invalid consent_time_field values, the Resource Server MUST exclude them from time-bounded queries.

Response:

{
  "stream": "conversations",
  "records_accepted": 2,
  "records_rejected": 0
}

Sync state

GET  /v1/state/{connector_id}
PUT  /v1/state/{connector_id}
Authorization: Bearer <owner_token>

These endpoints read and update the StreamState map. The optional grant_id query parameter addresses the state namespace for one continuous grant. If it is absent, the endpoint addresses the connector's global archival state.

GET response:

{
  "object": "stream_state",
  "connector_id": "https://registry.pdpp.dev/connectors/spotify-browser",
  "grant_id": "grt_8f72a1b3",
  "state": {
    "top_artists": { "last_updated": "2026-04-01T00:00:00Z" }
  },
  "updated_at": "2026-04-06T15:00:00Z"
}

If grant_id is absent from the request, it is omitted from the response. If it is present, the response contains only that grant's state namespace.

PUT request body:

{
  "state": {
    "top_artists": { "last_updated": "2026-04-06T15:00:00Z" }
  }
}

PUT returns the same shape as GET.

single_use runs MUST NOT read or persist grant-scoped state. The runtime sends state: null and discards any emitted STATE checkpoints for those runs.

Concurrent collection

Multiple runs for the same connector may execute concurrently. Record writes MUST be idempotent by primary key. A repeated write with the same key and data has no additional effect. The Resource Server MUST accept a cursor only when it advances the addressed global or grant-scoped state namespace. A slower run cannot regress that state.

Errors

Collection transport errors use the Core structured error envelope.

5. Conformance

Conformance claims for Core Resource Servers, Collection Resource Servers, connector runtimes, and connectors are independent. A combined product may claim more than one role, but Core conformance does not require parsing the Collection extension or implementing this profile.

A conformant Collection Resource Server:

1. Implements POST /v1/ingest/{stream} with owner authentication and the validation rules above.
2. Implements GET /v1/state/{connector_id} and PUT /v1/state/{connector_id}, including grant_id scoping for continuous runs.
3. Applies state updates monotonically within each addressed namespace.
4. Publishes Core freshness metadata on stream listings, stream metadata, and record-list responses, using status: "unknown" when recency is not known.

A conformant connector:

A conformant connector:

1. Reads START from stdin before emitting any messages.
2. Emits only valid JSONL messages as defined in this profile.
3. Emits DONE as the final message in all cases (including failures where possible).
4. Emits STATE periodically for streams that support incremental sync.
5. Does not store secrets (credentials, OTP codes) in STATE.
6. Does not emit INTERACTION while in waiting_for_interaction.
7. Treats missing required bindings as fatal errors.
8. Exits with status 0 on succeeded, non-zero on failed or cancelled.
9. Emits RECORD messages only within the scope provided in START: no undeclared streams, no records outside declared resources or time_range, and no extra top-level fields when fields is present.
10. If it cannot honor a declared resources, time_range, or fields constraint, emits an explicit SKIP_RESULT or fails the run; it never silently broadens scope.

A conformant connector runtime:

1. Performs binding matching before spawning the connector process.
2. Sends START as the first and only START message.
3. Handles INTERACTION messages by prompting the user or agent and sending INTERACTION_RESPONSE.
4. Sends INTERACTION_RESPONSE with status: "timeout" if no response arrives within timeout_seconds.
5. Persists STATE only after preceding records are durably written.
6. Does NOT persist STATE on failed or cancelled runs.
7. Uses the connector's global state namespace for proactive runs, the grant_id-scoped namespace for continuous grant runs, and state: null for single_use runs.
8. Terminates the connector process on protocol violations.
9. Does not log or persist credential data from INTERACTION_RESPONSE.
10. Sends an explicit non-empty scope in START. For grant-driven runs, this scope is a normalized, possibly narrowed projection of the grant and MUST NOT include wildcard stream names.
11. For grant-driven runs, never constructs a scope broader than the grant permits.
12. Rejects or discards connector emissions that fall outside the declared scope before durable write.
13. Terminates an active grant-driven run as soon as practical after learning that the grant was revoked.

6. TypeScript Types

type InteractionKind = 'credentials' | 'otp' | 'manual_action';
type StreamState = Record<string, Record<string, unknown>>;
type TimeRange = { since?: string; until?: string };
type CollectionStream = {
  name: string;
  resources?: string[];
  time_range?: TimeRange;
  fields?: string[];
};
type CollectionScope = {
  streams: CollectionStream[];
};

type RuntimeMessage =
  | {
      type: 'START';
      run_id: string;
      collection_mode: 'full_refresh' | 'incremental';
      scope: CollectionScope;
      state: StreamState | null;
      bindings: Record<string, Record<string, unknown>>;
    }
  | {
      type: 'INTERACTION_RESPONSE';
      request_id: string;
      status: 'success' | 'cancelled' | 'timeout';
      data?: Record<string, unknown>;
    };

type ConnectorMessage =
  | {
      type: 'RECORD';
      stream: string;
      key: string | string[];
      data: Record<string, unknown>;
      emitted_at: string;
      op?: 'upsert' | 'delete';
    }
  | {
      type: 'STATE';
      stream: string;
      cursor: Record<string, unknown>;
    }
  | {
      type: 'INTERACTION';
      request_id: string;
      kind: InteractionKind;
      message: string;
      schema?: Record<string, unknown>;
      timeout_seconds?: number;
    }
  | {
      type: 'SKIP_RESULT';
      stream?: string;
      reason?: string;
      message?: string;
    }
  | {
      type: 'PROGRESS';
      stream?: string;
      message: string;
      count?: number;
      total?: number;
    }
  | {
      type: 'DONE';
      status: 'succeeded' | 'failed' | 'cancelled';
      records_emitted: number;
      error?: { message: string; retryable: boolean };
    };