Package cn.net.pap.common.datastructure.tokenization

Class BPETokenization

java.lang.Object

cn.net.pap.common.datastructure.tokenization.BPETokenization

public class BPETokenization
extends Object

Constructor Summary

Constructors

Constructor	Description
BPETokenization()

Method Summary

Modifier and Type	Method	Description
Set<String>	getVocab()
List<String>	tokenize(String text)
void	train(List<String> words)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BPETokenization

public BPETokenization()

Method Details

train

public void train(List<String> words)

tokenize

public List<String> tokenize(String text)

getVocab

public Set<String> getVocab()