neuron compiling a bert model

0

Hi, I want to neuron compile a bert large model(patentbert from google) which has sequence length 512. How do I do this? Also I want to call the model as before or need to know what I should change while calling it. I compiled the anferico/bert-for-patents using the instructions here https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/huggingface_bert/huggingface_bert.html. Also is it the same to use patentbert from google and huggingface anferico/bert-for-patents ??

The saved model cli output for

  1. Google's patentbert MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']: The given SavedModel SignatureDef contains the following input(s): inputs['input_ids'] tensor_info: dtype: DT_INT64 shape: (-1, 512) name: input_ids:0 inputs['input_mask'] tensor_info: dtype: DT_INT64 shape: (-1, 512) name: input_mask:0 inputs['mlm_positions'] tensor_info: dtype: DT_INT64 shape: (-1, 45) name: mlm_positions:0 inputs['segment_ids'] tensor_info: dtype: DT_INT64 shape: (-1, 512) name: segment_ids:0 The given SavedModel SignatureDef contains the following output(s): outputs['cls_token'] tensor_info: dtype: DT_FLOAT shape: (-1, 1024) name: Squeeze:0 outputs['encoder_layer'] tensor_info: dtype: DT_FLOAT shape: (-1, 512, 1024) name: bert/encoder/layer_23/output/LayerNorm/batchnorm/add_1:0 outputs['mlm_logits'] tensor_info: dtype: DT_FLOAT shape: (-1, 39859) name: cls/predictions/BiasAdd:0 outputs['next_sentence_logits'] tensor_info: dtype: DT_FLOAT shape: (-1, 2) name: cls/seq_relationship/BiasAdd:0 Method name is: tensorflow/serving/predict

MetaGraphDef with tag-set: 'serve, tpu' contains the following SignatureDefs:

signature_def['serving_default']: The given SavedModel SignatureDef contains the following input(s): inputs['input_ids'] tensor_info: dtype: DT_INT64 shape: (-1, 512) name: input_ids:0 inputs['input_mask'] tensor_info: dtype: DT_INT64 shape: (-1, 512) name: input_mask:0 inputs['mlm_positions'] tensor_info: dtype: DT_INT64 shape: (-1, 45) name: mlm_positions:0 inputs['segment_ids'] tensor_info: dtype: DT_INT64 shape: (-1, 512) name: segment_ids:0 The given SavedModel SignatureDef contains the following output(s): outputs['cls_token'] tensor_info: dtype: DT_FLOAT shape: unknown_rank name: TPUPartitionedCall:5 outputs['encoder_layer'] tensor_info: dtype: DT_FLOAT shape: unknown_rank name: TPUPartitionedCall:6 outputs['mlm_logits'] tensor_info: dtype: DT_FLOAT shape: unknown_rank name: TPUPartitionedCall:4 outputs['next_sentence_logits'] tensor_info: dtype: DT_FLOAT shape: unknown_rank name: TPUPartitionedCall:7 Method name is: tensorflow/serving/predict

  1. Neuron compiled anferico/bert-for-patents MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']: The given SavedModel SignatureDef contains the following input(s): The given SavedModel SignatureDef contains the following output(s): outputs['__saved_model_init_op'] tensor_info: dtype: DT_INVALID shape: unknown_rank name: NoOp Method name is:

signature_def['serving_default']: The given SavedModel SignatureDef contains the following input(s): inputs['input_1'] tensor_info: dtype: DT_INT32 shape: (-1, 512) name: serving_default_input_1:0 inputs['input_2'] tensor_info: dtype: DT_INT32 shape: (-1, 512) name: serving_default_input_2:0 The given SavedModel SignatureDef contains the following output(s): outputs['output_1'] tensor_info: dtype: DT_FLOAT shape: (-1, 2) name: StatefulPartitionedCall:0 Method name is: tensorflow/serving/predict

Concrete Functions: Function Name: 'call' Option #1 Callable with: Argument #1 DType: list Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]

Function Name: '_default_save_signature' Option #1 Callable with: Argument #1 DType: list Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]

Function Name: 'aws_neuron_function' Option #1 Callable with: Argument #1 args_0 Argument #2 args_0_1

Function Name: 'call_and_return_all_conditional_losses' Option #1 Callable with: Argument #1 DType: list Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]

This is the stack trace when I replace the original bert model with neuron compiled model. Traceback (most recent call last): File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1645, in _call_with_flat_signature args.append(kwargs.pop(compat.as_str(keyword))) KeyError: 'input_1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1615, in _call_impl cancellation_manager) File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1651, in _call_with_flat_signature raise TypeError(f"{self._flat_signature_summary()} missing required " TypeError: signature_wrapper(input_1, input_2) missing required arguments: input_1, input_2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "main.py", line 319, in <module> verbose) File "main.py", line 223, in process_each_project verbose=verbose) File "/home/ubuntu/ranking_pipeline/rank_utils.py", line 437, in rank response, inputs, _ = self.model.predict(search_sentences) File "/home/ubuntu/ranking_pipeline/bert_utils.py", line 292, in predict inputs['mlm_ids'], dtype=tf.int64), File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1601, in call return self._call_impl(args, kwargs) File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1617, in _call_impl raise structured_err File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1611, in _call_impl cancellation_manager) File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1688, in _call_with_structured_signature self._structured_signature_check_missing_args(args, kwargs) File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1707, in _structured_signature_check_missing_args raise TypeError(f"{self._structured_signature_summary()} missing " TypeError: signature_wrapper(*, input_2, input_1) missing required arguments: input_1, input_2.

Thanks in advance Warm regards Ajay

asked 21 days ago78 views
1 Answer
0

Due to the 512 length limit, you have basically two options:

  1. You cut the longer texts off and only use the first 512 Tokens. The original BERT implementation (and probably the others as well) truncates longer sequences automatically. For most cases, this option is sufficient.
  2. You can split your text in multiple subtexts, classify each of them and combine the results back together ( choose the class which was predicted for most of the subtexts for example). This option is obviously more expensive.

You can read about these options here.

is it the same to use patentbert from google and huggingface anferico/bert-for-patents ?

Yes, both versions will work with the given use case. You can look at a simplified comparison between the services here.

profile picture
answered 16 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions