modeling

class BigBirdModel(num_layers, vocab_size, nhead, attn_dropout=0.1, dim_feedforward=3072, activation='gelu', normalize_before=False, block_size=1, window_size=3, num_global_blocks=1, num_rand_blocks=2, seed=None, pad_token_id=0, hidden_size=768, hidden_dropout_prob=0.1, max_position_embeddings=512, type_vocab_size=2, **kwargs)

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

forward(input_ids, token_type_ids=None, attention_mask_list=None, rand_mask_idx_list=None)

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters

• *inputs (tuple) – unpacked tuple arguments

• **kwargs (dict) – unpacked dict arguments

class BigBirdPretrainedModel(name_scope=None, dtype='float32')

Bases: paddlenlp.transformers.model_utils.PretrainedModel

An abstract class for pretrained BigBird models. It provides BigBird related model_config_file, resource_files_names, pretrained_resource_files_map, pretrained_init_configuration, base_model_prefix for downloading and loading pretrained models. See PretrainedModel for more details.

init_weights(layer)

Initialization hook

base_model_class

alias of paddlenlp.transformers.bigbird.modeling.BigBirdModel

class BigBirdForPretraining(bigbird)

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

forward(input_ids, token_type_ids=None, position_ids=None, rand_mask_idx_list=None, masked_positions=None)

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters

• *inputs (tuple) – unpacked tuple arguments

• **kwargs (dict) – unpacked dict arguments

class BigBirdPretrainingCriterion(vocab_size, use_nsp=False, ignore_index=0)

Bases: paddle.fluid.dygraph.layers.Layer

forward(prediction_scores, seq_relationship_score, masked_lm_labels, next_sentence_labels, masked_lm_scale, masked_lm_weights)

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters

• *inputs (tuple) – unpacked tuple arguments

• **kwargs (dict) – unpacked dict arguments

class BigBirdForSequenceClassification(bigbird, num_classes=None)

Bases: paddlenlp.transformers.bigbird.modeling.BigBirdPretrainedModel

forward(input_ids, token_type_ids=None, attention_mask_list=None, rand_mask_idx_list=None)

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters

• *inputs (tuple) – unpacked tuple arguments

• **kwargs (dict) – unpacked dict arguments

class BigBirdPretrainingHeads(hidden_size, vocab_size, activation, embedding_weights=None)

Bases: paddle.fluid.dygraph.layers.Layer

forward(sequence_output, pooled_output, masked_positions=None)

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters

• *inputs (tuple) – unpacked tuple arguments

• **kwargs (dict) – unpacked dict arguments