File: //usr/local/lib/python3.7/site-packages/charset_normalizer/__pycache__/md.cpython-37.pyc
B
���gDN � @ sb d dl mZ d dlmZ d dlmZ ddlmZmZm Z ddl
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZ G dd� d�Z G d d
� d
e �Z G d d � d e �Z G d
d� de �Z G dd� de �Z G dd� de �Z!G dd� de �Z"G dd� de �Z#G dd� de �Z$G dd� de �Z%edd �d d d d �d d!��Z&ed"d �d+d%d&d d&d'�d(d)��Z'd*S ),� )� annotations)� lru_cache)� getLogger� )� COMMON_SAFE_ASCII_CHARACTERS�TRACE� UNICODE_SECONDARY_RANGE_KEYWORD)�is_accentuated� is_arabic�is_arabic_isolated_form�is_case_variable�is_cjk� is_emoticon� is_hangul� is_hiragana� is_katakana�is_latin�is_punctuation� is_separator� is_symbol�is_thai�is_unprintable�
remove_accent�
unicode_rangec @ sP e Zd ZdZddd�dd�Zddd�dd �Zdd
�d d �Zed
d
�dd��ZdS )�MessDetectorPluginzy
Base abstract class used for mess detection plugins.
All detectors MUST extend and implement given methods.
�str�bool)� character�returnc C s t �dS )z@
Determine if given character should be fed in.
N)�NotImplementedError)�selfr � r! �?/usr/local/lib/python3.7/site-packages/charset_normalizer/md.py�eligible& s zMessDetectorPlugin.eligible�Nonec C s t �dS )z�
The main routine to be executed upon character.
Insert the logic in witch the text would be considered chaotic.
N)r )r r r! r! r" �feed, s zMessDetectorPlugin.feed)r c C s t �dS )zB
Permit to reset the plugin to the initial state.
N)r )r r! r! r" �reset3 s zMessDetectorPlugin.reset�floatc C s t �dS )z�
Compute the chaos ratio based on what your feed() has seen.
Must NOT be lower than 0.; No restriction gt 0.
N)r )r r! r! r" �ratio9 s zMessDetectorPlugin.ratioN) �__name__�
__module__� __qualname__�__doc__r# r% r& �propertyr( r! r! r! r" r s
r c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d
d �Zdd�d d
�Zedd�dd��ZdS )� TooManySymbolOrPunctuationPluginr$ )r c C s" d| _ d| _d| _d | _d| _d S )Nr F)�_punctuation_count�
_symbol_count�_character_count�_last_printable_charZ_frenzy_symbol_in_word)r r! r! r" �__init__C s
z)TooManySymbolOrPunctuationPlugin.__init__r r )r r c C s |� � S )N)� isprintable)r r r! r! r" r# K s z)TooManySymbolOrPunctuationPlugin.eligiblec C sp | j d7 _ || jkrf|tkrft|�r8| jd7 _n.|�� dkrft|�rft|�dkrf| jd7 _|| _d S )Nr F� ) r1 r2 r r r/ �isdigitr r r0 )r r r! r! r" r% N s
z%TooManySymbolOrPunctuationPlugin.feedc C s d| _ d| _d| _d S )Nr )r/ r1 r0 )r r! r! r" r& ` s z&TooManySymbolOrPunctuationPlugin.resetr' c C s0 | j dkrdS | j| j | j }|dkr,|S dS )Nr g g333333�?)r1 r/ r0 )r Zratio_of_punctuationr! r! r" r( e s
z&TooManySymbolOrPunctuationPlugin.ratioN) r) r* r+ r3 r# r% r& r- r( r! r! r! r" r. B s
r. c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d
d �Zdd�d d
�Zedd�dd��ZdS )�TooManyAccentuatedPluginr$ )r c C s d| _ d| _d S )Nr )r1 �_accentuated_count)r r! r! r" r3 r s z!TooManyAccentuatedPlugin.__init__r r )r r c C s |� � S )N)�isalpha)r r r! r! r" r# v s z!TooManyAccentuatedPlugin.eligiblec C s( | j d7 _ t|�r$| jd7 _d S )Nr )r1 r r8 )r r r! r! r" r% y s z TooManyAccentuatedPlugin.feedc C s d| _ d| _d S )Nr )r1 r8 )r r! r! r" r& s z TooManyAccentuatedPlugin.resetr' c C s* | j dk rdS | j| j }|dkr&|S dS )N� g gffffff�?)r1 r8 )r Zratio_of_accentuationr! r! r" r( � s
z TooManyAccentuatedPlugin.ratioN) r) r* r+ r3 r# r% r& r- r( r! r! r! r" r7 q s
r7 c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d
d �Zdd�d d
�Zedd�dd��ZdS )�UnprintablePluginr$ )r c C s d| _ d| _d S )Nr )�_unprintable_countr1 )r r! r! r" r3 � s zUnprintablePlugin.__init__r r )r r c C s dS )NTr! )r r r! r! r" r# � s zUnprintablePlugin.eligiblec C s( t |�r| jd7 _| jd7 _d S )Nr )r r< r1 )r r r! r! r" r% � s zUnprintablePlugin.feedc C s
d| _ d S )Nr )r< )r r! r! r" r& � s zUnprintablePlugin.resetr' c C s | j dkrdS | jd | j S )Nr g r: )r1 r< )r r! r! r" r( � s
zUnprintablePlugin.ratioN) r) r* r+ r3 r# r% r& r- r( r! r! r! r" r; � s
r; c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d
d �Zdd�d d
�Zedd�dd��ZdS )� SuspiciousDuplicateAccentPluginr$ )r c C s d| _ d| _d | _d S )Nr )�_successive_countr1 �_last_latin_character)r r! r! r" r3 � s z(SuspiciousDuplicateAccentPlugin.__init__r r )r r c C s |� � ot|�S )N)r9 r )r r r! r! r" r# � s z(SuspiciousDuplicateAccentPlugin.eligiblec C st | j d7 _ | jd k rjt|�rjt| j�rj|�� rJ| j�� rJ| jd7 _t|�t| j�krj| jd7 _|| _d S )Nr )r1 r? r �isupperr> r )r r r! r! r" r% � s
z$SuspiciousDuplicateAccentPlugin.feedc C s d| _ d| _d | _d S )Nr )r> r1 r? )r r! r! r" r& � s z%SuspiciousDuplicateAccentPlugin.resetr' c C s | j dkrdS | jd | j S )Nr g r5 )r1 r> )r r! r! r" r( � s
z%SuspiciousDuplicateAccentPlugin.ratioN) r) r* r+ r3 r# r% r& r- r( r! r! r! r" r= � s
r= c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d
d �Zdd�d d
�Zedd�dd��ZdS )�SuspiciousRanger$ )r c C s d| _ d| _d | _d S )Nr )�"_suspicious_successive_range_countr1 �_last_printable_seen)r r! r! r" r3 � s zSuspiciousRange.__init__r r )r r c C s |� � S )N)r4 )r r r! r! r" r# � s zSuspiciousRange.eligiblec C sx | j d7 _ |�� s&t|�s&|tkr0d | _d S | jd krD|| _d S t| j�}t|�}t||�rn| jd7 _|| _d S )Nr )r1 �isspacer r rC r � is_suspiciously_successive_rangerB )r r �unicode_range_a�unicode_range_br! r! r" r% � s
zSuspiciousRange.feedc C s d| _ d| _d | _d S )Nr )r1 rB rC )r r! r! r" r& � s zSuspiciousRange.resetr' c C s" | j dkrdS | jd | j }|S )N�
g r5 )r1 rB )r Z ratio_of_suspicious_range_usager! r! r" r( � s
zSuspiciousRange.ratioN) r) r* r+ r3 r# r% r& r- r( r! r! r! r" rA � s
rA c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d
d �Zdd�d d
�Zedd�dd��ZdS )�SuperWeirdWordPluginr$ )r c C s@ d| _ d| _d| _d| _d| _d| _d| _d| _d| _d| _ d S )Nr F� )
� _word_count�_bad_word_count�_foreign_long_count�_is_current_word_bad�_foreign_long_watchr1 �_bad_character_count�_buffer�_buffer_accent_count�_buffer_glyph_count)r r! r! r" r3 � s z SuperWeirdWordPlugin.__init__r r )r r c C s dS )NTr! )r r r! r! r" r#
s z SuperWeirdWordPlugin.eligiblec C s� |� � r�| j|7 _t|�r,| jd7 _| jdkr�t|�dksJt|�r�t|�dkr�t|�dkr�t|�dkr�t |�dkr�t
|�dkr�d| _t|�s�t|�s�t|�s�t |�s�t
|�r�| j d7 _ d S | js�d S |� � s�t
|�s�t|��r\| j�r\| jd7 _t| j�}| j|7 _|dk�r�| j| dk�r:d| _npt| jd ��r�| jd �� �r�tdd� | jD ��dk�r�| jd7 _d| _n | j dk�r�d| _| jd7 _|d k�r| j�rd
d � t| jtd |��D �}d}|�r�t|�| d
k�r�d}|�s| jd7 _d| _| j�rB| jd7 _| jt| j�7 _d| _d| _d| _d | _d | _ n6|dk�r�|�� dk�r�t|��r�d| _| j|7 _d S )Nr FT� g �?���c s s | ]}|� � V qd S )N)r@ )�.0�_r! r! r" � <genexpr>7 s z,SuperWeirdWordPlugin.feed.<locals>.<genexpr>� c S s g | ]\}}|� � r|�qS r! )r@ )rV �c�ir! r! r" �
<listcomp>@ s z-SuperWeirdWordPlugin.feed.<locals>.<listcomp>r g333333�?rJ > �-�>�<�=�|�~rW ) r9 rQ r rR rO r r
r r r r rS rD r r rK �lenr1 rN r@ �allrM �zip�rangerL rP r6 r )r r Z
buffer_lengthZcamel_case_dstZprobable_camel_casedr! r! r" r%
sv
zSuperWeirdWordPlugin.feedc C s4 d| _ d| _d| _d| _d| _d| _d| _d| _d S )NrJ Fr )rQ rN rO rL rK r1 rP rM )r r! r! r" r& ^ s zSuperWeirdWordPlugin.resetr' c C s$ | j dkr| jdkrdS | j| j S )N�
r g )rK rM rP r1 )r r! r! r" r( h s zSuperWeirdWordPlugin.ratioN) r) r* r+ r3 r# r% r& r- r( r! r! r! r" rI � s
Q
rI c @ s^ e Zd ZdZdd�dd�Zddd�d d
�Zddd�d d �Zdd�d
d�Zedd�dd��Z dS )�CjkInvalidStopPluginu�
GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and
can be easily detected. Searching for the overuse of '丅' and '丄'.
r$ )r c C s d| _ d| _d S )Nr )�_wrong_stop_count�_cjk_character_count)r r! r! r" r3 v s z CjkInvalidStopPlugin.__init__r r )r r c C s dS )NTr! )r r r! r! r" r# z s z CjkInvalidStopPlugin.eligiblec C s4 |dkr| j d7 _ d S t|�r0| jd7 _d S )N> � 丄� 丅r )ri r
rj )r r r! r! r" r% } s
zCjkInvalidStopPlugin.feedc C s d| _ d| _d S )Nr )ri rj )r r! r! r" r& � s zCjkInvalidStopPlugin.resetr' c C s | j dk rdS | j| j S )N� g )rj ri )r r! r! r" r( � s
zCjkInvalidStopPlugin.ratioN)
r) r* r+ r, r3 r# r% r& r- r( r! r! r! r" rh p s rh c @ sZ e Zd Zdd�dd�Zddd�dd �Zddd�d
d �Zdd�d d
�Zedd�dd��ZdS )�ArchaicUpperLowerPluginr$ )r c C s. d| _ d| _d| _d| _d| _d | _d| _d S )NFr T)�_buf� _character_count_since_last_sep� _successive_upper_lower_count�#_successive_upper_lower_count_finalr1 �_last_alpha_seen�_current_ascii_only)r r! r! r" r3 � s z ArchaicUpperLowerPlugin.__init__r r )r r c C s dS )NTr! )r r r! r! r" r# � s z ArchaicUpperLowerPlugin.eligiblec C s$ |� � ot|�}|dk}|r�| jdkr�| jdkrV|�� dkrV| jdkrV| j| j7 _d| _d| _d | _d| _| j d7 _ d| _d S | jdkr�|�
� dkr�d| _| jd k r�|� � r�| j� � s�|� � r�| j� � r�| jdkr�| jd7 _d| _q�d| _nd| _| j d7 _ | jd7 _|| _d S )NFr �@ r Tr5 )
r9 r rp r6 rt rr rq rs ro r1 �isasciir@ �islower)r r Z is_concernedZ chunk_sepr! r! r" r% � s8
z ArchaicUpperLowerPlugin.feedc C s. d| _ d| _d| _d| _d | _d| _d| _d S )Nr FT)r1 rp rq rr rs ro rt )r r! r! r" r& � s z ArchaicUpperLowerPlugin.resetr' c C s | j dkrdS | j| j S )Nr g )r1 rr )r r! r! r" r( � s
z ArchaicUpperLowerPlugin.ratioN) r) r* r+ r3 r# r% r& r- r( r! r! r! r" rn � s
* rn c @ sZ e Zd Zdd�dd�Zdd�dd�Zddd �d
d �Zddd �d d
�Zedd�dd��ZdS )�ArabicIsolatedFormPluginr$ )r c C s d| _ d| _d S )Nr )r1 �_isolated_form_count)r r! r! r" r3 � s z!ArabicIsolatedFormPlugin.__init__c C s d| _ d| _d S )Nr )r1 ry )r r! r! r" r& � s z ArabicIsolatedFormPlugin.resetr r )r r c C s t |�S )N)r
)r r r! r! r" r# � s z!ArabicIsolatedFormPlugin.eligiblec C s( | j d7 _ t|�r$| jd7 _d S )Nr )r1 r ry )r r r! r! r" r% � s z ArabicIsolatedFormPlugin.feedr' c C s | j dk rdS | j| j }|S )Nr: g )r1 ry )r Zisolated_form_usager! r! r" r( � s
z ArabicIsolatedFormPlugin.ratioN) r) r* r+ r3 r& r# r% r- r( r! r! r! r" rx � s
rx i )�maxsizez
str | Noner )rF rG r c C s� | dks|dkrdS | |kr dS d| kr4d|kr4dS d| ksDd|krHdS d| ksXd|krld| kshd|krldS | � d�|� d� }}x"|D ]}|tkr�q�||kr�dS q�W | dk|dk }}|s�|r�d | ks�d |kr�dS |r�|r�dS d
| ks�d
|k�r"d | k�sd |k�r
dS | d k�s |d k�r"dS d | k�sJd |k�sJ| d k�r�|d k�r�d
| k�s^d
|k�rbdS d| k�svd|k�rzdS | d k�s�|d k�r�dS dS )za
Determine if two Unicode range seen next to each other can be considered as suspicious.
NTFZLatinZ EmoticonsZ Combining� )�Hiragana�Katakana�CJK�Hangulz Basic Latin)r} r| � PunctuationZForms)�splitr )rF rG Zkeywords_range_aZkeywords_range_b�elZrange_a_jp_charsZrange_b_jp_charsr! r! r" rE � sR
rE i 皙�����?Fr r' )�decoded_sequence�maximum_threshold�debugr c C sb dd� t �� D �}t| �d }d}|dk r0d}n|dkr>d}nd }x|t| d
t|��D ]f\}}x |D ]} | �|�rd| �|� qdW |d kr�|| d ks�||d krVtd d
� |D ��}||krVP qVW |�rXtd�}
|
� t
d|� d|� d|� �� t| �dk�r.|
� t
d| dd� � �� |
� t
d| dd� � �� x(|D ] } |
� t
| j � d| j � �� �q4W t
|d�S )zw
Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier.
c S s g | ]
}|� �qS r! r! )rV Zmd_classr! r! r" r\ J s z mess_ratio.<locals>.<listcomp>r g i � i ru � �
r c s s | ] }|j V qd S )N)r( )rV �dtr! r! r" rX ` s z mess_ratio.<locals>.<genexpr>Zcharset_normalizerzIMess-detector extended-analysis start. intermediary_mean_mess_ratio_calc=z mean_mess_ratio=z maximum_threshold=rm zStarting with: Nz
Ending with: i�z: � )r �__subclasses__rc re rf r# r% �sumr �logr � __class__r( �round) r� r� r� Z detectors�lengthZmean_mess_ratioZ!intermediary_mean_mess_ratio_calcr �index�detector�loggerr� r! r! r" �
mess_ratioA s8
r� N)r� F)(�
__future__r � functoolsr �loggingr Zconstantr r r �utilsr r
r r r
r r r r r r r r r r r r r r. r7 r; r= rA rI rh rn rx rE r� r! r! r! r" �<module> s" L"/%1v L J