neuron compiling a bert model

Question

Hi, 
I want to neuron compile a bert large model(patentbert from google) which has sequence length 512. How do I do this?
Also I want to call the model as before or need to know what I should change while calling it. 
I compiled the anferico/bert-for-patents using the instructions here https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/huggingface_bert/huggingface_bert.html.
Also is it the same to use patentbert from google and huggingface anferico/bert-for-patents ??

The saved model cli output for 
1.  Google's patentbert
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_ids'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 512)
        name: input_ids:0
    inputs['input_mask'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 512)
        name: input_mask:0
    inputs['mlm_positions'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 45)
        name: mlm_positions:0
    inputs['segment_ids'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 512)
        name: segment_ids:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['cls_token'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1024)
        name: Squeeze:0
    outputs['encoder_layer'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 512, 1024)
        name: bert/encoder/layer_23/output/LayerNorm/batchnorm/add_1:0
    outputs['mlm_logits'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 39859)
        name: cls/predictions/BiasAdd:0
    outputs['next_sentence_logits'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 2)
        name: cls/seq_relationship/BiasAdd:0
  Method name is: tensorflow/serving/predict

MetaGraphDef with tag-set: 'serve, tpu' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_ids'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 512)
        name: input_ids:0
    inputs['input_mask'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 512)
        name: input_mask:0
    inputs['mlm_positions'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 45)
        name: mlm_positions:0
    inputs['segment_ids'] tensor_info:
        dtype: DT_INT64
        shape: (-1, 512)
        name: segment_ids:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['cls_token'] tensor_info:
        dtype: DT_FLOAT
        shape: unknown_rank
        name: TPUPartitionedCall:5
    outputs['encoder_layer'] tensor_info:
        dtype: DT_FLOAT
        shape: unknown_rank
        name: TPUPartitionedCall:6
    outputs['mlm_logits'] tensor_info:
        dtype: DT_FLOAT
        shape: unknown_rank
        name: TPUPartitionedCall:4
    outputs['next_sentence_logits'] tensor_info:
        dtype: DT_FLOAT
        shape: unknown_rank
        name: TPUPartitionedCall:7
  Method name is: tensorflow/serving/predict

2. Neuron compiled anferico/bert-for-patents
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_1'] tensor_info:
        dtype: DT_INT32
        shape: (-1, 512)
        name: serving_default_input_1:0
    inputs['input_2'] tensor_info:
        dtype: DT_INT32
        shape: (-1, 512)
        name: serving_default_input_2:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output_1'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 2)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict

Concrete Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          DType: list
          Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]

Function Name: '_default_save_signature'
    Option #1
      Callable with:
        Argument #1
          DType: list
          Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]

Function Name: 'aws_neuron_function'
    Option #1
      Callable with:
        Argument #1
          args_0
        Argument #2
          args_0_1

Function Name: 'call_and_return_all_conditional_losses'
    Option #1
      Callable with:
        Argument #1
          DType: list
          Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]

This is the stack trace when I replace the original bert model with neuron compiled model.
Traceback (most recent call last):
  File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1645, in _call_with_flat_signature
    args.append(kwargs.pop(compat.as_str(keyword)))
KeyError: 'input_1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1615, in _call_impl
    cancellation_manager)
  File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1651, in _call_with_flat_signature
    raise TypeError(f"{self._flat_signature_summary()} missing required "
TypeError: signature_wrapper(input_1, input_2) missing required arguments: input_1, input_2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 319, in 
    verbose)
  File "main.py", line 223, in process_each_project
    verbose=verbose)
  File "/home/ubuntu/ranking_pipeline/rank_utils.py", line 437, in rank
    response, inputs, _ = self.model.predict(search_sentences)
  File "/home/ubuntu/ranking_pipeline/bert_utils.py", line 292, in predict
    inputs['mlm_ids'], dtype=tf.int64),
  File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1601, in __call__
    return self._call_impl(args, kwargs)
  File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1617, in _call_impl
    raise structured_err
  File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1611, in _call_impl
    cancellation_manager)
  File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1688, in _call_with_structured_signature
    self._structured_signature_check_missing_args(args, kwargs)
  File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1707, in _structured_signature_check_missing_args
    raise TypeError(f"{self._structured_signature_summary()} missing "
TypeError: signature_wrapper(*, input_2, input_1) missing required arguments: input_1, input_2.

Thanks in advance
Warm regards
Ajay

Answer

**Due to the 512 length limit, you have basically two options:**

1. You cut the longer texts off and only use the first 512 Tokens. The original BERT implementation (and probably the others as well) truncates longer sequences automatically. For most cases, this option is sufficient.
2. You can split your text in multiple subtexts, classify each of them and combine the results back together ( choose the class which was predicted for most of the subtexts for example). This option is obviously more expensive.

You can read about these options [here.](https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification)

> is it the same to use patentbert from google and huggingface anferico/bert-for-patents ?

Yes, both versions will work with the given use case. You can look at a simplified comparison between the services [here.](https://levelup.gitconnected.com/bert-for-patents-by-google-vs-deeppatent-and-patentbert-simplified-2ccea99201bd)

neuron compiling a bert model

Contenus pertinents