File: //opt/osm/venv/lib/python3.10/site-packages/charset_normalizer/__pycache__/utils.cpython-310.pyc
o
=i�/ � @ s� d dl mZ d dlZd dlZd dlZd dlmZ d dlmZ d dl m
Z
d dl m Z d dl
mZ d dlmZ d d
lmZmZmZmZmZmZmZ e
ed �didd��Ze
ed �djdd��Ze
ed �dkdd��Ze
ed �didd��Z e
ed �didd��Z e
ed �didd ��Z e
ed �did d ��Z e
ed �did d ��Z e
ed �did!d"��Z!e
ed �did#d$��Z"e
ed �did%d&��Z#e
ed �did'd(��Z$e
ed �did)d*��Z%e
ed �did+d,��Z&e
ed �did-d.��Z'e
ed �did/d0��Z(e
ed �did1d2��Z)e
e*e�d �dld4d5��Z+e
ed �did6d7��Z,dmdnd=d>�Z-e
d?d �dodAdB��Z.dpdDdE�Z/dqdGdH�Z0drdsdLdM�Z1dtdQdR�Z2dudSdT�Z3dUej4dVfdvdZd[�Z5 dwdxdgdh�Z6dS )y� )� annotationsN)�IncrementalDecoder)�aliases)� lru_cache)�findall)� Generator)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION�COMMON_CJK_CHARACTERS)�maxsize� character�str�return�boolc C sd zt �| �}W n
ty Y dS w d|v p1d|v p1d|v p1d|v p1d|v p1d|v p1d|v p1d |v S )
NFz
WITH GRAVEz
WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz
WITH TILDEz WITH MACRONzWITH RING ABOVE�� unicodedata�name�
ValueError�r � description� r �F/opt/osm/venv/lib/python3.10/site-packages/charset_normalizer/utils.py�is_accentuated s( ��������r c C s. t �| �}|s | S |�d�}tt|d d��S )N� r � )r �
decomposition�split�chr�int)r Z
decomposed�codesr r r �
remove_accent- s
r& �
str | Nonec C s. t | �}t�� D ] \}}||v r| S qdS )zK
Retrieve the Unicode range official name from a single character.
N)�ordr
�items)r Z
character_ord�
range_nameZ ord_ranger r r �
unicode_range8 s �r+ c C �* z
t �| �}W d|v S ty Y dS w )NFZLATINr r r r r �is_latinF s ��r- c C s2 t �| �}d|v r dS t| �}|d u rdS d|v S )N�PTF� Punctuation�r �categoryr+ �r �character_category�character_ranger r r �is_punctuationO s
r5 c C sB t �| �}d|v s
d|v rdS t| �}|d u rdS d|v o |dkS )N�S�NTFZFormsZLor0 r2 r r r � is_symbol^ s
r8 c C s$ t | �}|d u r
dS d|v pd|v S )NFZ EmoticonsZ Pictographs)r+ )r r4 r r r � is_emoticonm s r9 c C s. | � � s| dv r
dS t�| �}d|v p|dv S )N> �<�>�+u ｜T�Z> ZPcZPdZPo)�isspacer r1 )r r3 r r r � is_separatorw s
r? c C s | � � | �� kS �N)�islower�isupper�r r r r �is_case_variable� s rD c C r, )NFZCJKr �r Zcharacter_namer r r �is_cjk� � ��rF c C r, )NFZHIRAGANAr rE r r r � is_hiragana� rG rH c C r, )NFZKATAKANAr rE r r r � is_katakana� rG rI c C r, )NFZHANGULr rE r r r � is_hangul� rG rJ c C r, )NFZTHAIr rE r r r �is_thai� rG rK c C r, )NF�ARABICr rE r r r � is_arabic� rG rM c C s4 zt �| �}W n
ty Y dS w d|v od|v S )NFrL z
ISOLATED FORMr rE r r r �is_arabic_isolated_form� s �rN c C s | t vS r@ )r rC r r r �is_cjk_uncommon� s rO r* c s t � fdd�tD ��S )Nc 3 s � | ]}|� v V qd S r@ r )�.0�keyword�r* r r � <genexpr>� s � z-is_unicode_range_secondary.<locals>.<genexpr>)�anyr rR r rR r �is_unicode_range_secondary� s rU c C s( | � � du o| �� du o| dko| dkS )NF�u )r> � isprintablerC r r r �is_unprintable� s
���rX � �sequence�bytes� search_zoner$ c C s� t | t�st�t| �}tt| dt||�� jddd��}t|�dkr$dS |D ]'}|�� � dd�}t
� � D ]\}}||krB| S ||krL| S q4q&dS )zW
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
N�ascii�ignore��errorsr �-�_) �
isinstancer[ � TypeError�lenr r �min�decode�lower�replacer r) )rZ r\ Zseq_len�resultsZspecified_encoding�encoding_alias�
encoding_ianar r r �any_specified_encoding� s&
� ��rm � r c C s | dv pt t�d| � ��jt�S )zQ
Verify is a specific encoding is a multi byte one based on it IANA name
> �utf_8� utf_16_leZ utf_8_sig�utf_16� utf_16_be�utf_32� utf_32_be�utf_7� utf_32_le�
encodings.)�
issubclass� importlib�
import_moduler r )r r r r �is_multi_byte_encoding s
��r{ �tuple[str | None, bytes]c C sJ t D ] }t | }t|t�r|g}|D ]}| �|�r!||f S qqdS )z9
Identify and extract SIG/BOM in given sequence.
)N� )r
rc r[ �
startswith)rZ �
iana_encodingZmarks�markr r r �identify_sig_or_bom s
��r� r c C s | dvS )N> rs rq r )r r r r �should_strip_sig_or_bom( s r� T�cp_name�strictc C sN | � � �dd�} t�� D ]\}}| ||fv r| S q |r%td| � d���| S )zIReturns the Python normalized encoding name (Not the IANA official name).ra rb z Unable to retrieve IANA for '�')rh ri r r) r )r� r� rk rl r r r � iana_name, s �r� � iana_name_a� iana_name_b�floatc C s� t | �st |�r
dS t�d| � ��j}t�d|� ��j}|dd�}|dd�}d}td�D ]}t|g�}|�|�|�|�krA|d7 }q,|d S ) Ng rw r^ r_ r � r � )r{ ry rz r �ranger[ rg ) r� r� Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_count�iZ
to_be_decodedr r r �
cp_similarity= s
�r� c C s | t v o |t | v S )z�
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
)r )r� r� r r r �
is_cp_similarQ s
�r� Zcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s�level�
format_string�Nonec C s: t �| �}|�|� t �� }|�t �|�� |�|� d S r@ )�logging� getLogger�setLevel�
StreamHandler� setFormatter� Formatter�
addHandler)r r� r� �logger�handlerr r r �set_logging_handler\ s
r� � sequencesrl �offsetsr� �
chunk_size�bom_or_sig_available�strip_sig_or_bom� sig_payload�is_multi_byte_decoder�decoded_payload�Generator[str, None, None]c c s& � |r |du r |D ]} || | | � }
|
s d S |
V q d S |D ]p} | | } | t | �d kr/q | | | | � } |rA|du rA|| } | j||rHdndd�}
|r�| dkr�t|d�}
|r�|
d |
� |vr�t| | d d �D ]#}| || � } |r{|du r{|| } | j|dd�}
|
d |
� |v r� nqi|
V q d S )
NF� r^ r� r_ r r � ���)re rg rf r� )r� rl r� r� r� r� r� r� r� r� �chunkZ chunk_endZ cut_sequenceZchunk_partial_size_chk�jr r r �cut_sequence_chunksi sD � �
�
� ��r� )r r r r )r r r r )r r r r' )r* r r r )rY )rZ r[ r\ r$ r r' )r r r r )rZ r[ r r| )r r r r )T)r� r r� r r r )r� r r� r r r� )r� r r� r r r )r r r� r$ r� r r r� r@ )r� r[ rl r r� r� r� r$ r� r r� r r� r[ r� r r� r' r r� )7�
__future__r ry r� r �codecsr Zencodings.aliasesr � functoolsr �rer �typingr Z_multibytecodecr Zconstantr
r r r
r r r r r& r+ r- r5 r8 r9 r? rD rF rH rI rJ rK rM rN rO re rU rX rm r{ r� r� r� r� r� �INFOr� r� r r r r �<module> s~ $
��