# Pipeline data (pdata)

<!-- status autogenerated section -->
| Status        |           |
| ------------- |-----------|
| Stability     | [stable]: traces, metrics, logs   |
| Issues        | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector?query=is%3Aissue%20is%3Aopen%20label%3Apkg%2Fpdata%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aopen+is%3Aissue+label%3Apkg%2Fpdata) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector?query=is%3Aissue%20is%3Aclosed%20label%3Apkg%2Fpdata%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aclosed+is%3Aissue+label%3Apkg%2Fpdata) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner)    | [@bogdandrutu](https://www.github.com/bogdandrutu), [@dmitryax](https://www.github.com/dmitryax) |

[stable]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stable
<!-- end autogenerated section -->

Pipeline data (pdata) implements data structures that represent telemetry data in-memory. All data received
is converted into this format, travels through the pipeline in this format, and is converted from this format by
exporters when sending.

Current implementation primarily uses OTLP protobuf structs as the underlying data structures for many of the
declared structs. We keep a pointer to OTLP protobuf in the "orig" member field. This allows efficient translation
to/from OTLP wire protocol. The underlying data structure is kept private so that we are free to make changes to it
in the future.

The pdata API is designed to avoid mutable data sharing and bugs that stem from that. Each pdata instance cannot
contain a reference to an object that is used in another pdata instance.

## API naming convention

### Package names

Names of pdata packages don't follow names of the protobuf packages. The pdata has a package per telemetry type
starting with `p`, e.g. `ptrace`, and `pcommon` package which includes pdata API for protobuf definitions from
`common` and `resource` protobuf packages.

### Protobuf message representation in pdata

Pipeline data structs SHOULD be based on the names of the underlying OTLP protobuf messages. Data types for
protobuf messages defined as part of another message SHOULD include the owner's name as a prefix. The following
examples are two pdata structs based on protobuf messages defined at the package level and as part of another
message:

- `pmetric.NumberDataPoint` based on a `NumberDataPoint` protobuf message defined in
  [metrics.proto](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto)
  package.
- `pmetric.ExponentialHistogramDataPointBuckets` based on a `Buckets` protobuf message defined in
  `ExponentialHistogramDataPoint` message of
  [metrics.proto](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto)
  package.

Exceptions to the naming rules are possible, but not encouraged. Another name can be chosen for brevity or if the
struct provides a different interface to manipulate the underlying data. The following exceptions are currently
accepted:

- `plog.Logs` based on `LogsData` protobuf message.
- `pmetric.Metrics` based on `MetricsData` protobuf message.
- `ptrace.Traces` based on `TracesData` protobuf message.
- `pcommon.Slice` based on `ArrayValue` protobuf message.
- `pcommon.Map` based on `KeyValueList` protobuf message.
- `pcommon.Value` based on `AnyValue` protobuf message.

Each pdata struct MUST have an initialization function starting with `New` prefix. Usage of zero-initialized values is
prohibited and can cause a panic. Each pdata struct MUST provide the following methods:

- `MoveTo(<type>)`: moves all the data from one struct instance to another. The destination instance is overwritten and the
  source instance is re-initialized as a new empty instance.
- `CopyTo(<type>)`: deep copies all the data from one struct instance to another. The destination instance is
  overwritten and the source instance is not modified.

Each pdata struct based on a protobuf message SHOULD have getter methods for every protobuf field. Exceptions to
this rule are allowed if exposing a field is not desirable in pdata. For example, a deprecated protobuf field MAY
not be exposed in pdata. Also, pdata MAY provide a different interface to manipulate protobuf fields without
exposing them explicitly as is done with `pcommon.Map`.

### Protobuf fields representation in pdata

#### Singular fields

Name of a pdata getter methods representing singular protobuf fields SHOULD match the name the protobuf field but
written in CamelCase aligned with Go naming conventions. If a protobuf field is of scalar or enum type, the
corresponding pdata struct MUST provide a setter method with `Set` prefix.

Some fields of scalar type in a protobuf message MAY be represented by another pdata type providing an additional
interface, for example `pcommon.Timestamp` is another type that wraps `uint64` value and provides an additional
interface to work with timestamps. pdata fields returning `pcommon.Timestamp` don't follow the recommended naming
schema, `Timestamp` word is used instead of `TimeUnixNano` to represent protobuf fields that contain
`time_unix_nano`, for example `StartTimestamp` pdata field is used for `start_time_unix_nano` protobuf fields.

#### Optional fields

Each optional field in a protobuf message exposed in pdata MUST have the following additional methods:
- A method starting with `Has` to determine if the field is set. For example,
  `func (ms HistogramDataPoint) HasMin() bool`.
- A method starting with `Remove` prefix to remove a value associated with the optional field. For example,
  `func (ms HistogramDataPoint) RemoveMin()`.

#### OneOf fields

A pdata struct representing a protobuf message with an exposed oneof field MUST provide getter and setter methods for
each option of the oneof field. Name of each setter and getter SHOULD be called the same as the protobuf oneof field
options. The following exception is adopted in pdata for numeric oneof protobuf fields for brevity:

```protobuf
oneof value {
  double as_double = 4;
  sfixed64 as_int = 6;
}
```

is represented in pdata in a shorter form of the following methods:

```golang
DoubleValue() float64
SetDoubleValue(float64)
IntValue() int64
SetIntValue(v int64)
```

If a oneof field option is another protobuf message, the setter name MUST include `Empty` in its name. Such setter
MUST set the oneof field to an empty initialized struct and return it. The following conditions define whether the
name of the oneof field must be included in the setter and getter method names:

1. If a oneof field represents the whole protobuf message, name of the oneof field itself MAY be omitted in setter
   and getter methods. For example:

```go
func (ms Metric) Sum() Sum
func (ms Metric) SetEmptySum() Sum
```

2. If a oneof field is relevant only to a particular field of the message, the pdata getter and setter methods MUST
   include name of the oneof field along with name of the option. For example:

```go
func (ms NumberDataPoint) IntValue() int64
func (ms NumberDataPoint) SetIntValue(v int64)
```

Additionally, the pdata struct MUST provide a type getter for every exposed oneof protobuf field. Name of the getter
MUST be `Type` for oneof fields representing the whole protobuf message (1), or `<oneof field>Type` for oneof fields
relevant only to a particular field. The function MUST return an `int32` enum type called `<struct name>Type` for (1)
and `<struct name><oneof field>Type` for (2). For example:

- `func (ms Metric) Type() MetricType` (1)
- `func (ms NumberDataPoint) ValueType() NumberDataPointValueType` (2)

The enum constants MUST be called the same as the type with a suffix named after the oneof option, e.g.:

- `MetricTypeSum` (1)
- `NumberDataPointValueTypeString` (2)

An unset oneof protobuf value MUST be represented by a pdata constant with `Empty` suffix, e.g.:

- `MetricTypeEmpty` (1)
- `NumberDataPointValueTypeEmpty` (2)

The pdata enum type SHOULD have `String()` method returning a string value of the corresponding oneof field option.

#### Repeated protobuf fields

Each repeated protobuf message field exposed in pdata MUST be represented as another pdata struct that SHOULD be
called the same as the underlying protobuf field with `Slice` suffix. An exception example is `pdata.Map` that
provides a different interface to manipulate the underlying protobuf data.

#### Repeated scalar fields

Each repeated scalar protobuf field exposed in pdata MUST be represented as another pdata type wrapping a native go
slice. Name of the type SHOULD include name of the primitive data type ending with `Slice` suffix, e.g. `UInt64Slice`.

#### Enum fields

Each protobuf enum field exposed in pdata MUST have a type declared with underlying `int64` type. Name of the type
SHOULD follow the same rules as for struct type names representing protobuf messages:

- If a protobuf enum is defined on the package level, pdata type SHOULD have the same name.
- If a protobuf enum is defined as part of another message, pdata type SHOULD include the protobuf message name as a
  prefix.

Constants defined for the pdata int64 type MUST have the same names and numeric values defined in protobuf. Names
of the constants must be translated from ALL_CAPS_SNAKE_CASE to CamelCase.

The pdata enum type SHOULD have `String()` method returning a string value for each constant.

Example of a protobuf enum definition in pdata:

```golang
type AggregationTemporality int32
const (
 AggregationTemporalityUnspecified = AggregationTemporality(otlpmetrics.AggregationTemporality_AGGREGATION_TEMPORALITY_UNSPECIFIED)
 AggregationTemporalityDelta = AggregationTemporality(otlpmetrics.AggregationTemporality_AGGREGATION_TEMPORALITY_DELTA)
 AggregationTemporalityCumulative = AggregationTemporality(otlpmetrics.AggregationTemporality_AGGREGATION_TEMPORALITY_CUMULATIVE)
)
```

### Flags

Flags fields are typically defined in protobuf as `uint32` fields representing 32 distinct boolean flags. The
exposed protobuf flags fields MUST be defined as new types with `uint32` underlying type and called according to
the same rules as struct names representing protobuf messages and other enum types. Each pdata flags type MUST have
an empty variable of the fields type with `Default` prefix. Each flag MUST have a getter method returning a
particular flag and a setter method with `With` prefix returning a new instance of the pdata flags type. Any instance
of the pdata type is immutable since it's just a wrapper over `uint32`.

The following pdata type is used for log record flags field:

```golang
type LogRecordFlags uint32
var DefaultLogRecordFlags = LogRecordFlags(0)
func (ms LogRecordFlags) IsSampled() bool
func (ms LogRecordFlags) WithIsSampled(b bool) LogRecordFlags 
```

### Examples

The following protobuf message:

```protobuf
message NumberDataPoint {
  reserved 1;
  repeated opentelemetry.proto.common.v1.KeyValue attributes = 7;
  fixed64 start_time_unix_nano = 2;
  fixed64 time_unix_nano = 3;
  oneof value {
    double as_double = 4;
    sfixed64 as_int = 6;
  }
  repeated Exemplar exemplars = 5;
  uint32 flags = 8;
}
```

is represented by the following pdata API

```go
type NumberDataPoint
func NewNumberDataPoint() NumberDataPoint 
func (ms NumberDataPoint) MoveTo(dest NumberDataPoint)
func (ms NumberDataPoint) CopyTo(dest NumberDataPoint)
func (ms NumberDataPoint) Attributes() pcommon.Map
func (ms NumberDataPoint) StartTimestamp() pcommon.Timestamp
func (ms NumberDataPoint) SetStartTimestamp(v pcommon.Timestamp)
func (ms NumberDataPoint) Timestamp() pcommon.Timestamp
func (ms NumberDataPoint) SetTimestamp(v pcommon.Timestamp)
func (ms NumberDataPoint) ValueType() NumberDataPointValueType
func (ms NumberDataPoint) DoubleValue() float64
func (ms NumberDataPoint) SetDoubleValue(v float64)
func (ms NumberDataPoint) IntValue() int64
func (ms NumberDataPoint) SetIntValue(v int64)
func (ms NumberDataPoint) Exemplars() ExemplarSlice 
func (ms NumberDataPoint) Flags() DataPointFlags
func (ms NumberDataPoint) SetFlags(v DataPointFlags)
```