Skip to content

encoder

Code for encoding protocol message primitives.

Contains the logic for encoding every logical protocol field type into one of the 5 physical wire types.

This code is designed to push the Python interpreter's performance to the limits.

The basic idea is that at startup time, for every field (i.e. every FieldDescriptor) we construct two functions: a "sizer" and an "encoder". The sizer takes a value of this field's type and computes its byte size. The encoder takes a writer function and a value. It encodes the value into byte strings and invokes the writer function to write those strings. Typically the writer function is the write() method of a BytesIO.

We try to do as much work as possible when constructing the writer and the sizer rather than when calling them. In particular: * We copy any needed global functions to local variables, so that we do not need to do costly global table lookups at runtime. * Similarly, we try to do any attribute lookups at startup time if possible. * Every field's tag is encoded to bytes at startup, since it can't change at runtime. * Whatever component of the field size we can compute at startup, we do. * We avoid sharing code if doing so would make the code slower and not sharing does not burden us too much. For example, encoders for repeated fields do not just call the encoders for singular fields in a loop because this would add an extra function call overhead for every loop iteration; instead, we manually inline the single-value encoder into the loop. * If a Python function lacks a return statement, Python actually generates instructions to pop the result of the last statement off the stack, push None onto the stack, and then return that. If we really don't care what value is returned, then we can save two instructions by returning the result of the last statement. It looks funny but it helps. * We assume that type and bounds checking has happened at a higher level.

`BoolEncoder(field_number, is_repeated, is_packed)`

Returns an encoder for a boolean field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def BoolEncoder(field_number, is_repeated, is_packed):
  """Returns an encoder for a boolean field."""

  false_byte = b'\x00'
  true_byte = b'\x01'
  if is_packed:
    tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
    local_EncodeVarint = _EncodeVarint
    def EncodePackedField(write, value, deterministic):
      write(tag_bytes)
      local_EncodeVarint(write, len(value), deterministic)
      for element in value:
        if element:
          write(true_byte)
        else:
          write(false_byte)
    return EncodePackedField
  elif is_repeated:
    tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_VARINT)
    def EncodeRepeatedField(write, value, unused_deterministic=None):
      for element in value:
        write(tag_bytes)
        if element:
          write(true_byte)
        else:
          write(false_byte)
    return EncodeRepeatedField
  else:
    tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_VARINT)
    def EncodeField(write, value, unused_deterministic=None):
      write(tag_bytes)
      if value:
        return write(true_byte)
      return write(false_byte)
    return EncodeField

`BytesEncoder(field_number, is_repeated, is_packed)`

Returns an encoder for a bytes field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def BytesEncoder(field_number, is_repeated, is_packed):
  """Returns an encoder for a bytes field."""

  tag = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
  local_EncodeVarint = _EncodeVarint
  local_len = len
  assert not is_packed
  if is_repeated:
    def EncodeRepeatedField(write, value, deterministic):
      for element in value:
        write(tag)
        local_EncodeVarint(write, local_len(element), deterministic)
        write(element)
    return EncodeRepeatedField
  else:
    def EncodeField(write, value, deterministic):
      write(tag)
      local_EncodeVarint(write, local_len(value), deterministic)
      return write(value)
    return EncodeField

`BytesSizer(field_number, is_repeated, is_packed)`

Returns a sizer for a bytes field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def BytesSizer(field_number, is_repeated, is_packed):
  """Returns a sizer for a bytes field."""

  tag_size = _TagSize(field_number)
  local_VarintSize = _VarintSize
  local_len = len
  assert not is_packed
  if is_repeated:
    def RepeatedFieldSize(value):
      result = tag_size * len(value)
      for element in value:
        l = local_len(element)
        result += local_VarintSize(l) + l
      return result
    return RepeatedFieldSize
  else:
    def FieldSize(value):
      l = local_len(value)
      return tag_size + local_VarintSize(l) + l
    return FieldSize

`GroupEncoder(field_number, is_repeated, is_packed)`

Returns an encoder for a group field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def GroupEncoder(field_number, is_repeated, is_packed):
  """Returns an encoder for a group field."""

  start_tag = TagBytes(field_number, wire_format.WIRETYPE_START_GROUP)
  end_tag = TagBytes(field_number, wire_format.WIRETYPE_END_GROUP)
  assert not is_packed
  if is_repeated:
    def EncodeRepeatedField(write, value, deterministic):
      for element in value:
        write(start_tag)
        element._InternalSerialize(write, deterministic)
        write(end_tag)
    return EncodeRepeatedField
  else:
    def EncodeField(write, value, deterministic):
      write(start_tag)
      value._InternalSerialize(write, deterministic)
      return write(end_tag)
    return EncodeField

`GroupSizer(field_number, is_repeated, is_packed)`

Returns a sizer for a group field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def GroupSizer(field_number, is_repeated, is_packed):
  """Returns a sizer for a group field."""

  tag_size = _TagSize(field_number) * 2
  assert not is_packed
  if is_repeated:
    def RepeatedFieldSize(value):
      result = tag_size * len(value)
      for element in value:
        result += element.ByteSize()
      return result
    return RepeatedFieldSize
  else:
    def FieldSize(value):
      return tag_size + value.ByteSize()
    return FieldSize

`MapEncoder(field_descriptor)`

Encoder for extensions of MessageSet.

Maps always have a wire format like this

message MapEntry { key_type key = 1; value_type value = 2; } repeated MapEntry map = N;

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def MapEncoder(field_descriptor):
  """Encoder for extensions of MessageSet.

  Maps always have a wire format like this:
    message MapEntry {
      key_type key = 1;
      value_type value = 2;
    }
    repeated MapEntry map = N;
  """
  # Can't look at field_descriptor.message_type._concrete_class because it may
  # not have been initialized yet.
  message_type = field_descriptor.message_type
  encode_message = MessageEncoder(field_descriptor.number, False, False)

  def EncodeField(write, value, deterministic):
    value_keys = sorted(value.keys()) if deterministic else value
    for key in value_keys:
      entry_msg = message_type._concrete_class(key=key, value=value[key])
      encode_message(write, entry_msg, deterministic)

  return EncodeField

`MapSizer(field_descriptor, is_message_map)`

Returns a sizer for a map field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def MapSizer(field_descriptor, is_message_map):
  """Returns a sizer for a map field."""

  # Can't look at field_descriptor.message_type._concrete_class because it may
  # not have been initialized yet.
  message_type = field_descriptor.message_type
  message_sizer = MessageSizer(field_descriptor.number, False, False)

  def FieldSize(map_value):
    total = 0
    for key in map_value:
      value = map_value[key]
      # It's wasteful to create the messages and throw them away one second
      # later since we'll do the same for the actual encode.  But there's not an
      # obvious way to avoid this within the current design without tons of code
      # duplication. For message map, value.ByteSize() should be called to
      # update the status.
      entry_msg = message_type._concrete_class(key=key, value=value)
      total += message_sizer(entry_msg)
      if is_message_map:
        value.ByteSize()
    return total

  return FieldSize

`MessageEncoder(field_number, is_repeated, is_packed)`

Returns an encoder for a message field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def MessageEncoder(field_number, is_repeated, is_packed):
  """Returns an encoder for a message field."""

  tag = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
  local_EncodeVarint = _EncodeVarint
  assert not is_packed
  if is_repeated:
    def EncodeRepeatedField(write, value, deterministic):
      for element in value:
        write(tag)
        local_EncodeVarint(write, element.ByteSize(), deterministic)
        element._InternalSerialize(write, deterministic)
    return EncodeRepeatedField
  else:
    def EncodeField(write, value, deterministic):
      write(tag)
      local_EncodeVarint(write, value.ByteSize(), deterministic)
      return value._InternalSerialize(write, deterministic)
    return EncodeField

`MessageSetItemEncoder(field_number)`

Encoder for extensions of MessageSet.

The message set message looks like this

message MessageSet { repeated group Item = 1 { required int32 type_id = 2; required string message = 3; } }

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def MessageSetItemEncoder(field_number):
  """Encoder for extensions of MessageSet.

  The message set message looks like this:
    message MessageSet {
      repeated group Item = 1 {
        required int32 type_id = 2;
        required string message = 3;
      }
    }
  """
  start_bytes = b"".join([
      TagBytes(1, wire_format.WIRETYPE_START_GROUP),
      TagBytes(2, wire_format.WIRETYPE_VARINT),
      _VarintBytes(field_number),
      TagBytes(3, wire_format.WIRETYPE_LENGTH_DELIMITED)])
  end_bytes = TagBytes(1, wire_format.WIRETYPE_END_GROUP)
  local_EncodeVarint = _EncodeVarint

  def EncodeField(write, value, deterministic):
    write(start_bytes)
    local_EncodeVarint(write, value.ByteSize(), deterministic)
    value._InternalSerialize(write, deterministic)
    return write(end_bytes)

  return EncodeField

`MessageSetItemSizer(field_number)`

Returns a sizer for extensions of MessageSet.

The message set message looks like this

message MessageSet { repeated group Item = 1 { required int32 type_id = 2; required string message = 3; } }

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def MessageSetItemSizer(field_number):
  """Returns a sizer for extensions of MessageSet.

  The message set message looks like this:
    message MessageSet {
      repeated group Item = 1 {
        required int32 type_id = 2;
        required string message = 3;
      }
    }
  """
  static_size = (_TagSize(1) * 2 + _TagSize(2) + _VarintSize(field_number) +
                 _TagSize(3))
  local_VarintSize = _VarintSize

  def FieldSize(value):
    l = value.ByteSize()
    return static_size + local_VarintSize(l) + l

  return FieldSize

`MessageSizer(field_number, is_repeated, is_packed)`

Returns a sizer for a message field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def MessageSizer(field_number, is_repeated, is_packed):
  """Returns a sizer for a message field."""

  tag_size = _TagSize(field_number)
  local_VarintSize = _VarintSize
  assert not is_packed
  if is_repeated:
    def RepeatedFieldSize(value):
      result = tag_size * len(value)
      for element in value:
        l = element.ByteSize()
        result += local_VarintSize(l) + l
      return result
    return RepeatedFieldSize
  else:
    def FieldSize(value):
      l = value.ByteSize()
      return tag_size + local_VarintSize(l) + l
    return FieldSize

`StringEncoder(field_number, is_repeated, is_packed)`

Returns an encoder for a string field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def StringEncoder(field_number, is_repeated, is_packed):
  """Returns an encoder for a string field."""

  tag = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
  local_EncodeVarint = _EncodeVarint
  local_len = len
  assert not is_packed
  if is_repeated:
    def EncodeRepeatedField(write, value, deterministic):
      for element in value:
        encoded = element.encode('utf-8')
        write(tag)
        local_EncodeVarint(write, local_len(encoded), deterministic)
        write(encoded)
    return EncodeRepeatedField
  else:
    def EncodeField(write, value, deterministic):
      encoded = value.encode('utf-8')
      write(tag)
      local_EncodeVarint(write, local_len(encoded), deterministic)
      return write(encoded)
    return EncodeField

`StringSizer(field_number, is_repeated, is_packed)`

Returns a sizer for a string field.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def StringSizer(field_number, is_repeated, is_packed):
  """Returns a sizer for a string field."""

  tag_size = _TagSize(field_number)
  local_VarintSize = _VarintSize
  local_len = len
  assert not is_packed
  if is_repeated:
    def RepeatedFieldSize(value):
      result = tag_size * len(value)
      for element in value:
        l = local_len(element.encode('utf-8'))
        result += local_VarintSize(l) + l
      return result
    return RepeatedFieldSize
  else:
    def FieldSize(value):
      l = local_len(value.encode('utf-8'))
      return tag_size + local_VarintSize(l) + l
    return FieldSize

`TagBytes(field_number, wire_type)`

Encode the given tag and return the bytes. Only called at startup.

Source code in client/ayon_hiero/vendor/google/protobuf/internal/encoder.py

def TagBytes(field_number, wire_type):
  """Encode the given tag and return the bytes.  Only called at startup."""

  return bytes(_VarintBytes(wire_format.PackTag(field_number, wire_type)))