File: //usr/local/lib/python3.7/site-packages/charset_normalizer/__pycache__/utils.cpython-37.pyc
B
���g�. � @ s. d dl mZ d dlZd dlZd dlZd dlmZ d dlmZ d dl m
Z
d dl m Z d dl
mZ d dlmZ d d
lmZmZmZmZmZmZ e
ed �d d
d�dd��Ze
ed �d d d�dd��Ze
ed �d dd�dd��Ze
ed �d d
d�dd��Ze
ed �d d
d�dd��Z e
ed �d d
d�dd��Z e
ed �d d
d�d d ��Z e
ed �d d
d�d d ��Z e
ed �d d
d�d d!��Z e
ed �d d
d�d"d#��Z!e
ed �d d
d�d$d%��Z"e
ed �d d
d�d&d'��Z#e
ed �d d
d�d(d)��Z$e
ed �d d
d�d*d+��Z%e
ed �d d
d�d,d-��Z&e
ed �d d
d�d.d/��Z'e
e(e�d �d d
d0�d1d2��Z)e
ed �d d
d�d3d4��Z*d[d6d7dd8�d9d:�Z+e
d;d �d d
d<�d=d>��Z,d6d?d@�dAdB�Z-d d
dC�dDdE�Z.d\d d
d dG�dHdI�Z/d d dJdK�dLdM�Z0d d d
dK�dNdO�Z1dPej2dQfd d7d dRdS�dTdU�Z3d]d6d dVd7d
d
d6d
ddWdX�
dYdZ�Z4dS )^� )� annotationsN)�IncrementalDecoder)�aliases)� lru_cache)�findall)� Generator)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION)�maxsize�str�bool)� character�returnc C sd yt �| �}W n tk
r" dS X d|kpbd|kpbd|kpbd|kpbd|kpbd|kpbd|kpbd |kS )
NFz
WITH GRAVEz
WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz
WITH TILDEz WITH MACRONzWITH RING ABOVE)� unicodedata�name�
ValueError)r � description� r �B/usr/local/lib/python3.7/site-packages/charset_normalizer/utils.py�is_accentuated s r c C s. t �| �}|s| S |�d�}tt|d d��S )N� r � )r �
decomposition�split�chr�int)r Z
decomposed�codesr r r �
remove_accent, s
r# z
str | Nonec C s. t | �}x t�� D ]\}}||kr|S qW dS )zK
Retrieve the Unicode range official name from a single character.
N)�ordr
�items)r Z
character_ord�
range_nameZ ord_ranger r r �
unicode_range7 s
r' c C s, yt �| �}W n tk
r" dS X d|kS )NFZLATIN)r r r )r r r r r �is_latinE s
r( c C s2 t �| �}d|krdS t| �}|d kr*dS d|kS )N�PTF� Punctuation)r �categoryr' )r �character_category�character_ranger r r �is_punctuationN s
r. c C sB t �| �}d|ksd|kr dS t| �}|d kr2dS d|ko@|dkS )N�S�NTFZFormsZLo)r r+ r' )r r, r- r r r � is_symbol] s
r1 c C s$ t | �}|d krdS d|kp"d|kS )NFZ EmoticonsZ Pictographs)r' )r r- r r r � is_emoticonl s r2 c C s. | � � s| dkrdS t�| �}d|kp,|dkS )N> � ｜�+�<�>T�Z> �Pc�Pd�Po)�isspacer r+ )r r, r r r � is_separatorv s
r< c C s | � � | �� kS )N)�islower�isupper)r r r r �is_case_variable� s r? c C s, yt �| �}W n tk
r" dS X d|kS )NF�CJK)r r r )r �character_namer r r �is_cjk� s
rB c C s, yt �| �}W n tk
r" dS X d|kS )NFZHIRAGANA)r r r )r rA r r r � is_hiragana� s
rC c C s, yt �| �}W n tk
r" dS X d|kS )NFZKATAKANA)r r r )r rA r r r � is_katakana� s
rD c C s, yt �| �}W n tk
r" dS X d|kS )NFZHANGUL)r r r )r rA r r r � is_hangul� s
rE c C s, yt �| �}W n tk
r" dS X d|kS )NFZTHAI)r r r )r rA r r r �is_thai� s
rF c C s, yt �| �}W n tk
r" dS X d|kS )NF�ARABIC)r r r )r rA r r r � is_arabic� s
rH c C s4 yt �| �}W n tk
r" dS X d|ko2d|kS )NFrG z
ISOLATED FORM)r r r )r rA r r r �is_arabic_isolated_form� s
rI )r& r c s t � fdd�tD ��S )Nc 3 s | ]}|� kV qd S )Nr )�.0�keyword)r& r r � <genexpr>� s z-is_unicode_range_secondary.<locals>.<genexpr>)�anyr )r& r )r& r �is_unicode_range_secondary� s rN c C s( | � � dko&| �� dko&| dko&| dkS )NF�u )r; � isprintable)r r r r �is_unprintable� s rQ � �bytesr! )�sequence� search_zoner c C s� t | t�st�t| �}tt| dt||�� jddd��}t|�dkrHdS xJ|D ]B}|�� � dd�}x,t
� � D ] \}}||kr�|S ||krl|S qlW qNW dS )zW
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
N�ascii�ignore)�errorsr �-�_) �
isinstancerS � TypeError�lenr r �min�decode�lower�replacer r% )rT rU Zseq_len�resultsZspecified_encoding�encoding_alias�
encoding_ianar r r �any_specified_encoding� s
re � )r r c C s | dkp t t�d| � ��jt�S )zQ
Verify is a specific encoding is a multi byte one based on it IANA name
> � utf_8_sig�utf_8�utf_7�utf_32� utf_32_le� utf_32_be�utf_16� utf_16_be� utf_16_lez
encodings.)�
issubclass� importlib�
import_moduler r )r r r r �is_multi_byte_encoding� s
rs ztuple[str | None, bytes])rT r c C sJ xDt D ]<}t | }t|t�r"|g}x |D ]}| �|�r(||fS q(W qW dS )z9
Identify and extract SIG/BOM in given sequence.
)N� )r
r[ rS �
startswith)rT �
iana_encodingZmarks�markr r r �identify_sig_or_bom s
rx )rv r c C s | dkS )N> rm rj r )rv r r r �should_strip_sig_or_bom" s ry T)�cp_name�strictr c C sN | � � �dd�} x$t�� D ]\}}| ||gkr|S qW |rJtd| � d���| S )zIReturns the Python normalized encoding name (Not the IANA official name).rY rZ z Unable to retrieve IANA for '�')r` ra r r% r )rz r{ rc rd r r r � iana_name& s r} �float)� iana_name_a� iana_name_br c C s� t | �st |�rdS t�d| � ��j}t�d|� ��j}|dd�}|dd�}d}x6td�D ]*}t|g�}|�|�|�|�krZ|d7 }qZW |d S ) Ng z
encodings.rW )rX r � r � )rs rq rr r �rangerS r_ ) r r� Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_count�iZ
to_be_decodedr r r �
cp_similarity7 s
r� c C s | t ko|t | kS )z�
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
)r )r r� r r r �
is_cp_similarK s r� Zcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s�None)r �level�
format_stringr c C s: t �| �}|�|� t �� }|�t �|�� |�|� d S )N)�logging� getLogger�setLevel�
StreamHandler� setFormatter� Formatter�
addHandler)r r� r� �logger�handlerr r r �set_logging_handlerV s
r� r� zGenerator[str, None, None])
� sequencesrd �offsets�
chunk_size�bom_or_sig_available�strip_sig_or_bom� sig_payload�is_multi_byte_decoder�decoded_payloadr c c s0 |r8|dkr8x(|D ] } || | | � }
|
s,P |
V qW n�x�|D ]�} | | } | t | �d kr\q>| | | | � } |r�|dkr�|| } | j||r�dndd�}
|�r"| dk�r"t|d�}
|�r"|
d |
� |k�r"xXt| | d d �D ]D}| || � } |r�|dkr�|| } | j|dd�}
|
d |
� |kr�P q�W |
V q>W d S )
NF� rW r{ )rX r r � ���)r] r_ r^ r� )r� rd r� r� r� r� r� r� r� r� �chunkZ chunk_endZ cut_sequenceZchunk_partial_size_chk�jr r r �cut_sequence_chunksc s8
r� )rR )T)N)5�
__future__r rq r� r �codecsr Zencodings.aliasesr � functoolsr �rer �typingr Z_multibytecodecr Zconstantr
r r r
r r r r# r' r( r. r1 r2 r< r? rB rC rD rE rF rH rI r] rN rQ re rs rx ry r} r� r� �INFOr� r� r r r r �<module> sN