Hi,
I want to neuron compile a bert large model(patentbert from google) which has sequence length 512. How do I do this?
Also I want to call the model as before or need to know what I should change while calling it.
I compiled the anferico/bert-for-patents using the instructions here https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/huggingface_bert/huggingface_bert.html.
Also is it the same to use patentbert from google and huggingface anferico/bert-for-patents ??
The saved model cli output for
- Google's patentbert
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_ids'] tensor_info:
dtype: DT_INT64
shape: (-1, 512)
name: input_ids:0
inputs['input_mask'] tensor_info:
dtype: DT_INT64
shape: (-1, 512)
name: input_mask:0
inputs['mlm_positions'] tensor_info:
dtype: DT_INT64
shape: (-1, 45)
name: mlm_positions:0
inputs['segment_ids'] tensor_info:
dtype: DT_INT64
shape: (-1, 512)
name: segment_ids:0
The given SavedModel SignatureDef contains the following output(s):
outputs['cls_token'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 1024)
name: Squeeze:0
outputs['encoder_layer'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 512, 1024)
name: bert/encoder/layer_23/output/LayerNorm/batchnorm/add_1:0
outputs['mlm_logits'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 39859)
name: cls/predictions/BiasAdd:0
outputs['next_sentence_logits'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 2)
name: cls/seq_relationship/BiasAdd:0
Method name is: tensorflow/serving/predict
MetaGraphDef with tag-set: 'serve, tpu' contains the following SignatureDefs:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_ids'] tensor_info:
dtype: DT_INT64
shape: (-1, 512)
name: input_ids:0
inputs['input_mask'] tensor_info:
dtype: DT_INT64
shape: (-1, 512)
name: input_mask:0
inputs['mlm_positions'] tensor_info:
dtype: DT_INT64
shape: (-1, 45)
name: mlm_positions:0
inputs['segment_ids'] tensor_info:
dtype: DT_INT64
shape: (-1, 512)
name: segment_ids:0
The given SavedModel SignatureDef contains the following output(s):
outputs['cls_token'] tensor_info:
dtype: DT_FLOAT
shape: unknown_rank
name: TPUPartitionedCall:5
outputs['encoder_layer'] tensor_info:
dtype: DT_FLOAT
shape: unknown_rank
name: TPUPartitionedCall:6
outputs['mlm_logits'] tensor_info:
dtype: DT_FLOAT
shape: unknown_rank
name: TPUPartitionedCall:4
outputs['next_sentence_logits'] tensor_info:
dtype: DT_FLOAT
shape: unknown_rank
name: TPUPartitionedCall:7
Method name is: tensorflow/serving/predict
- Neuron compiled anferico/bert-for-patents
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['__saved_model_init_op']:
The given SavedModel SignatureDef contains the following input(s):
The given SavedModel SignatureDef contains the following output(s):
outputs['__saved_model_init_op'] tensor_info:
dtype: DT_INVALID
shape: unknown_rank
name: NoOp
Method name is:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_1'] tensor_info:
dtype: DT_INT32
shape: (-1, 512)
name: serving_default_input_1:0
inputs['input_2'] tensor_info:
dtype: DT_INT32
shape: (-1, 512)
name: serving_default_input_2:0
The given SavedModel SignatureDef contains the following output(s):
outputs['output_1'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 2)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
Concrete Functions:
Function Name: 'call'
Option #1
Callable with:
Argument #1
DType: list
Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]
Function Name: '_default_save_signature'
Option #1
Callable with:
Argument #1
DType: list
Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]
Function Name: 'aws_neuron_function'
Option #1
Callable with:
Argument #1
args_0
Argument #2
args_0_1
Function Name: 'call_and_return_all_conditional_losses'
Option #1
Callable with:
Argument #1
DType: list
Value: [TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_1'), TensorSpec(shape=(None, 512), dtype=tf.int32, name='input_2')]
This is the stack trace when I replace the original bert model with neuron compiled model.
Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1645, in _call_with_flat_signature
args.append(kwargs.pop(compat.as_str(keyword)))
KeyError: 'input_1'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1615, in _call_impl
cancellation_manager)
File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1651, in _call_with_flat_signature
raise TypeError(f"{self._flat_signature_summary()} missing required "
TypeError: signature_wrapper(input_1, input_2) missing required arguments: input_1, input_2.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 319, in <module>
verbose)
File "main.py", line 223, in process_each_project
verbose=verbose)
File "/home/ubuntu/ranking_pipeline/rank_utils.py", line 437, in rank
response, inputs, _ = self.model.predict(search_sentences)
File "/home/ubuntu/ranking_pipeline/bert_utils.py", line 292, in predict
inputs['mlm_ids'], dtype=tf.int64),
File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1601, in call
return self._call_impl(args, kwargs)
File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1617, in _call_impl
raise structured_err
File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1611, in _call_impl
cancellation_manager)
File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1688, in _call_with_structured_signature
self._structured_signature_check_missing_args(args, kwargs)
File "/home/ubuntu/venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1707, in _structured_signature_check_missing_args
raise TypeError(f"{self._structured_signature_summary()} missing "
TypeError: signature_wrapper(*, input_2, input_1) missing required arguments: input_1, input_2.
Thanks in advance
Warm regards
Ajay