hj: dZddlZddlmZmZddlmZmZddlmZddl m Z ddl m Z dd l m Zdd l mZejd Zgd ZGd dZdedejjjdefdZdedefdZGddZdS)z)[Experimental] Text Only Local Tokenizer.N)AnyIterable)OptionalUnion)sentencepiece_model_pb2)_common)_local_tokenizer_loader) _transformers)typeszgoogle_genai.local_tokenizer)_parse_hex_byte_token_str_to_bytesLocalTokenizer_TextsAccumulatorceZdZdZddZdeefdZdeej ddfdZ dej ddfd Z d ej ddfd Z d ejdejfd ZdeejddfdZdeejddfdZdejddfdZdejdejfdZdejdejfdZdeeefdeeeffdZdedefdZdS)raAccumulates countable texts from `Content` and `Tool` objects. This class is responsible for traversing complex `Content` and `Tool` objects and extracting all the text content that should be included when calculating token counts. A key feature of this class is its ability to detect unsupported fields in `Content` objects. If a user provides a `Content` object with fields that this local tokenizer doesn't recognize (e.g., new fields added in a future API update), this class will log a warning. The detection mechanism for `Content` objects works by recursively building a "counted" version of the input object. This "counted" object only contains the data that was successfully processed and added to the text list for tokenization. After traversing the input, the original `Content` object is compared to the "counted" object. If they don't match, it signifies the presence of unsupported fields, and a warning is logged. returnNcg|_dSN_textsselfs `/var/www/app.qaans.com/backend/venv/lib/python3.11/site-packages/google/genai/local_tokenizer.py__init__z_TextsAccumulator.__init__;s DKKKc|jSrrrs r get_textsz_TextsAccumulator.get_texts>s ;rcontentsc:|D]}||dSr) add_content)rrcontents r add_contentsz_TextsAccumulator.add_contentsAs2   w  rr!ctjg|j}|jr|jD]}|jJtj}|j|jtd|j |j|_|j &| |j |j |_ |j &| |j |j |_ |j +|j |_ |j|j |j||d|dkr#t"d|d|ddSdS)N)partsrolez6LocalTokenizers do not support non-text content types.T) exclude_nonezHContent contains unsupported types for token counting. Supported fields z. Got .)r Contentr%r$Part file_data inline_data ValueErrorvideo_metadata function_calladd_function_callfunction_responseadd_function_responsetextrappend model_dumploggerwarning)rr!counted_contentpart counted_parts rr z_TextsAccumulator.add_contentEsm"7<@@@O}3-33$$000z|| > %)9)EF      *(,(;, %   )  !3 4 4 4'+'9, $  ! -  $ $T%; < < <+/+A, ( 9 "i,  +  TY ' ' '$$\2222t,,0J0J1K11 nn 7$ 7 7,3 7 7 7rr.c|jr|j|jtj|j}|jr#||j}||_dSdS)zProcesses a function call and adds relevant text to the accumulator. Args: function_call: The function call to process. )nameN)r;rr3r FunctionCallargs_dict_traverse)rr.counted_function_call counted_argss rr/z#_TextsAccumulator.add_function_calldsy - k+,,,!.M4FGGG0(();<)rr0counted_function_responsecounted_responses rr1z'_TextsAccumulator.add_function_responses!& 6 8 8> k*/000'8'=$!<,,->-GHH+;(((<#>  D k-2333*>*C"'' k-9:::  *#.&C??+?+JKK0B"-$?)=)FGG.>"+ ''rschemactj}|jr |j|_|jr |j|_|j |j|_|jr+|j|j|j|_|jr+|j|j|j|_|j r+|j |j |j |_ |j r+|j |j |j |_ |j r |j |_ |j r!||j }||_ |jr\i}|j D]9\}}|j|||}|||<:||_|jr!||j}||_|S)zProcesses a schema and adds relevant text to the accumulator. Args: schema: The schema to process. Returns: The new schema object with only countable fields. )r Schematypetitledefaultformatrr3rWenumextendrequiredproperty_orderingitemsrY propertiesexample _any_traverse) rr[counted_schemacounted_schema_itemsdkeyvalue counted_valuecounted_schema_examples rrYz_TextsAccumulator.add_schemas\^^N {("Kn |*#\n ~!%~n }, k'''$mn 6 k+,,,#)#5n  {( k%%%"Kn 0 k))) &n B)/)An& |2!__V\::1n $ a)//11*#u 3.. #"#n ~6#11&.AA5n rrlci}|jt||D]\}}||||<|S)zProcesses a dict and adds relevant text to the accumulator. Args: d: The dict to process. Returns: The new dict object with only countable fields. )rrclistkeysrfri)rrl counted_dictrmvals rr>z _TextsAccumulator._dict_traversesiLKtAFFHH~~&&&GGII22S,,S11l3 rrnct|trj||St|tr|St|t rfd|DS|S)zProcesses a value and adds relevant text to the accumulator. Args: value: The value to process. Returns: The new value with only countable fields. c:g|]}|S)ri).0itemrs r z3_TextsAccumulator._any_traverse..s' 9 9 94d  && 9 9 9r) isinstancestrrr3dictr>rr)rrns` rriz_TextsAccumulator._any_traverses% k l E4    ' '' E4  9 9 9 95 9 9 99 lr)rN)__name__ __module__ __qualname____doc__rrr}rr r(r"r r<r/rDrIrLrQrOr1rVrEr]rYr~rr>rirxrrrr's &    # 8EM#: t    4> 0U-? 0D 0 0 0 0 5: %*    Xej1d4 ()? @4 4444 <$5 <  < < < <("'";(  (((((+u|+ ++++Z d38n c3h    rrtokenr^rc|tjjjjkr$t |ddS|dddS)Nrbig)length byteorderu▁ utf-8) r ModelProto SentencePieceTypeBYTEr to_bytesreplaceencode)rr^s rrrs_ $ / = B GGG 5 ! ! * *!u * E EE == $ $ + +G 4 44rclt|dkrtd||dr|dstd| t |ddd}n #t$rtd |wxYw|d krtd ||S) zParses a hex byte string of the form '<0xXX>' and returns the integer value. Raises ValueError if the input is malformed or the byte value is invalid. zInvalid byte length: z<0x>zInvalid byte format: zInvalid hex value: zByte value out of range: )lenr, startswithendswithint)rrus rr r s  ZZ1__ 4U44 5 55   % 6s(;(;6 4U44 5 554 eAaCj"  CC 444 2522 3 334 CZZ 888 9 99 *s #A<<BceZdZdZdefdZejddddee j e j fde e j d e jfd Zejddee j e j fd e jfd ZdS) ra[Experimental] Text Only Local Tokenizer. This class provides a local tokenizer for text only token counting. LIMITATIONS: - Only supports text based tokenization and no multimodal tokenization. - Forward compatibility depends on the open-source tokenizer models for future Gemini versions. - For token counting of tools and response schemas, the `LocalTokenizer` only supports `types.Tool` and `types.Schema` objects. Python functions or Pydantic models cannot be passed directly. model_namec$tj||_d|_|jtjvr tj|j|_dStj|j|_tj|j|_dSr) loaderget_tokenizer_name_tokenizer_name _model_protoGEMMA_TOKENIZER_TO_MODEL_NAMESget_huggingface_tokenizer _tokenizerload_model_protoget_sentencepiece)rrs rrzLocalTokenizer.__init__$sv!4Z@@DD vDDD89MNNdooo 1$2FGGd01EFFdooorzThe SDK's local tokenizer implementation is experimental and may change in the future. It only supports text based tokenization.N)configrrrctj|}t}tj|pi}|||jr||j|j r+|j j r| |j j |j r-|tj|j g|j t|}tjt%d|DS)aCounts the number of tokens in a given text. Args: contents: The contents to tokenize. config: The configuration for counting tokens. Returns: A `CountTokensResult` containing the total number of tokens. Usage: .. code-block:: python from google.genai import local_tokenizer tokenizer = local_tokenizer.LocalTokenizer(model_name='gemini-2.5-flash') result = tokenizer.count_tokens("What is your name?") print(result) # total_tokens=5 c34K|]}t|VdSr)r)rytokenss r z.LocalTokenizer.count_tokens..Vs(??V??????r) total_tokens)t t_contentsrr CountTokensConfigmodel_validater"rJrLgeneration_configresponse_schemarYsystem_instructionrrrrrCountTokensResultsum)rrrprocessed_contentstext_accumulator tokens_lists r count_tokenszLocalTokenizer.count_tokens-s1:h//(**  $ 3 3FLb A AF!!"4555 |/  ... LF$<$LL!!&":"JKKK  O##AL&2K1L$M$MNNN/((.>.H.H.J.J)K)KLLK  "??;?????   rc tj|}g}t}|D]B}|||jr$|jD]}||jCg}jtj vrj t| }t||D]Q\} } |tj| dj | D| Rtj|Sj | } t| |D]O\} } |tjd| jDfd| jD| Ptj|S)aBComputes the tokens ids and string pieces in the input. Args: contents: The contents to tokenize. Returns: A `ComputeTokensResult` containing the token information. Usage: .. code-block:: python from google.genai import local_tokenizer tokenizer = local_tokenizer.LocalTokenizer(model_name='gemini-2.5-flash') result = tokenizer.compute_tokens("What is your name?") print(result) # tokens_info=[TokensInfo(token_ids=[279, 329, 1313, 2508, 13], tokens=[b' What', b' is', b' your', b' name', b'?'], role='user')] cg|]?}|ddddd@S)_rrs▁ )rr)ryrs rr{z1LocalTokenizer.compute_tokens..sRMM#s++VG__W_d33r) token_idsrr%) tokens_infocg|] }|j Srx)id)rypieces rr{z1LocalTokenizer.compute_tokens..sCCCeCCCrcng|]1}t|jjj|jj2Srx)rrrpiecesrr^)ryrrs rr{z1LocalTokenizer.compute_tokens..sL&k4#4#;EH#E#Jr)rrrr r$r3r%rrrrrrrrzipr TokensInfoconvert_ids_to_tokensComputeTokensResultEncodeAsImmutableProtor) rrrrolesrr!r token_infosrtoken_idr% tokens_protos tokens_protos ` rcompute_tokenszLocalTokenizer.compute_tokensYs+4h// E(**%%%""7+++ % % %A ,,w| $ $ $ $K vDDD/((.>.H.H.J.J)K)KLLi 511   .(D  ""&!F!Fx!P!P       &; ? ? ??O::""$$M"-77   d  CC|/BCCC ,2         $ = = ==r)rrrrr}rr experimental_warningrr ContentListUnionContentListUnionDictrCountTokensConfigOrDictrrrrrxrrrrs"  GGGGG 7B9= &&&e,e.HHI&u45 &   &&& &P 7BE>e,e.HHIE>  E>E>E> E>E>E>rr)rloggingtypingrrrr sentencepiecerr r rr rr getLoggerr5__all__rr}rrrbytesrrr rrxrrrs 0/ """"""""111111//////   9 : :   MMMMMMMM`5 5-8FK5 5555 3 3    ,L>L>L>L>L>L>L>L>L>L>r