docs

Quick Start

Installation
Chinese sentiment analysis with PaddleNLP: A 10-Minute Tutorial

Data Preparation

Overview
Datasets List
Load Datasets
Customize Your Datasets
Data Preprocess

Tutorials

Text Classification
Word Embeddings
Semantic Matching
Text Generation
Machine Translation
Reading Comprehension
General Dialogue
Sequence Labeling
Lexical Analysis

Advanced Guide

Model Compression
Deployment
Distributed Training

Community Contribution

Dataset Contribution

API Reference

paddlenlp.data
- collate
- iterator
- sampler
- tokenizer
- vocab
paddlenlp.datasets
paddlenlp.embeddings
paddlenlp.ext_op
paddlenlp.layers
paddlenlp.metrics
paddlenlp.seq2vec
paddlenlp.transformers
paddlenlp.utils

PaddleNLP

»
paddlenlp.data »
tokenizer
Edit on GitHub

tokenizer¶

Next Previous

© Copyright 2021, PaddleNLP. Revision 44b08ad4.

Built with Sphinx using a theme provided by Read the Docs.

Read the Docs v: docs

Versions: latest; docs

Downloads

On Read the Docs: Project Home; Builds