File: //opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/autotracer.cpython-37.pyc
B
V�d�S �
@ s� d Z dZddlZddlZddlZddlmZ ddlmZm Z m
Z
ddl m Z m Z ddl
mZ ddlmZmZmZmZmZ dd lmZ ddlZdd
lmZ d d lm Z d d
l m Z m Z m Z ddl!m"Z"m#Z#m$Z$ ddl%m&Z& ddl'm(Z( ddl)m*Z* ddl+m,Z,m-Z-m.Z. dZ/yddl0m1Z2 W n, e3k
�rD Z4 z e4Z/dZ2W ddZ4[4X Y nX eG dd� d��Z5eG dd� d��Z6eG dd� d��Z7G dd� de�Z8e9e8d �d d �Z:G d d � d e �Z;e<d!k�r�e.� ej=d"ej>d#� e;� Z?e?� dS )$z'
This module contains Autotracer class
z ssa.modules� N)� defaultdict)� dataclass�field�asdict)�datetime� timedelta)� fnmatchcase)�Iterator�Optional�
NamedTuple�List�Iterable)�urlparse)�setup_database� )�Common)�iter_domains_data�iter_urls_data�get_url_durations� )�status�disabled_users�is_edition_supported)�
load_tunables)�autotracing_tunables_schema)�autotracing_stats_file)�is_xray_version_supported�is_kernel_version_supported� sentry_initc @ sJ e Zd ZU dZeed< eed< eed< eed< eed< eed< eed< d S )
�URLzURL data container�uri� avg_duration� max_duration�total_slow_reqs�
total_reqs�
belongs_to�belongs_to_userN)�__name__�
__module__� __qualname__�__doc__�str�__annotations__�int� r. r. �G/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/autotracer.pyr 2 s
r c @ s� e Zd ZU dZdZeed< dZeed< dZeed< dZ eed< dZ
eed< dZ eed< dZ eed < dZ
eed
< dZeed < dZeed < dZeed
< dS )� RulesRejectszRejects counters containerr �non_wp� throttled�density�slowness� max_slowness�disabled�nginx� frequency� server_limit� domain_limit� no_domainN)r' r( r) r* r1 r- r, r2 r3 r4 r5 r6 r7 r8 r9 r: r; r. r. r. r/ r0 > s
r0 c @ sD e Zd ZU dZeed< dZeed< dZeed< e e
d�Z e
ed< dS ) �Statsz AutoTracer statistics container�
rules_versionr �urls_processed�
urls_selected)�default_factory�rejectsN) r' r( r) r* r+ r, r> r- r? r r0 rA r. r. r. r/ r< N s
r< c @ s" e Zd ZU dZeed< eed< dS )�URLSz"
Representation of an URL
� domain_name�uri_pathN)r' r( r) r* r+ r, r. r. r. r/ rB W s
rB )�url�returnc C s\ t | �}|jrd|j� �nd}|jr2|j� |� �nd}|j�dd�}|�d�d }t||�S )z�
Split URL into domain_name and uripath including query string
:param url: URL of format protocol://domain/path;parameters?query#fragment
:return: namedtuple URL(domain_name, uripath)
�?� �/zwww.�:r )r �query�path�netloc�replace�splitrB )rE � fragments�qsr Z_no_www_netlocZ_no_port_netlocr. r. r/ � url_split_ s rR c s� e Zd ZdZd7� fdd� Z� fdd�Zeed�dd ��Zee e
e d
�d d ��Z e e
d
�dd�Zed�dd�Zd8e
e ed�dd�Ze e
d�dd�Zeee
d�dd�Zee e
d�d d �Zee e
d�d d �Ze
e ee
d �d!d"�Ze
e
e
d#�d$d%�Zee d�d&d'�Z d9e
e ee d(�d)d*�Z ee d�d+d,�Z e e d�d-d.�Z dd�d/d0�Z!dd�d1d2�Z"ed�d3d4�Z#ed�d5d6�Z$� Z%S ):�
AutoTracerz0
SSA autotracing module implementation.
Nc sN t � �� t�d�| _| j�dt� t� | _t� | _ d | _
|rB|nt � | _ d S )NZ auto_tracerzAutoTracer enabled: %s)
�super�__init__�logging� getLogger�logger�info� __package__�listr �
tasks_list�statsr �engine)�selfr^ )� __class__r. r/ rU r s
zAutoTracer.__init__c sF t � s| j�d� d S t� s,| j�d� d S t� sB| j�d� d S td krztj�d�rj| j� dt
t �� n | j�d� d S tj }|d k �r:| j�d� t
� �� | j�d| j� t| j�| _x|| �� D ]p}| j�d |j� | j jd
7 _y||j| jd � W q� tk
�r4 } z| j� d t
|�� W d d }~X Y q�X q�W | �� d S )
Nz*AutoTracer skipped: unsupported CL editionz.AutoTracer skipped: unsupported kernel versionz-AutoTracer skipped: unsupported X-Ray versionz/opt/alt/php-xrayz1AutoTracer skipped: X-ray module import error: %sz'AutoTracer skipped: X-Ray not installedzAutoTracer startedz AutoTracer loaded config: %szStarting auto task for %sr )rE Z
tracing_countzFailed to start task: %s)r rX rY r r �xray_lib�osrL �exists�errorr+ �xray_import_errorZstart_autotracingrT rU �debug�configr< r= r] �urls_scheduledr r? �request_number� Exception�save_iteration_stats)r_ Z
start_toolrE �e)r` r. r/ �__call__| s8
(zAutoTracer.__call__)rF c C s
t dt�S )z$
Load configuration
zautotracing.json)r r r. r. r. r/ � load_conf� s zAutoTracer.load_conf)rC rF c C s t dkr dS t �| �S )z;
Gets domain info for the specified domain
N)ra � domain_info)rC r. r. r/ �gets_domaininfo� s zAutoTracer.gets_domaininfo)� user_namerF c C s" | j r tdkrdS t�|�jS dS )z>
Says if nginx is enabled for a specific user
NF)Z
skip_nginxra ZNginxUserCache�
is_enabled)r_ rq r. r. r/ �nginx_is_enabled� s
zAutoTracer.nginx_is_enabledc C s, t dd� | jD ��}| j�d|� | j| S )z>Recalculate limit per server taking into account running tasksc S s( g | ] }|d dkr|d dkr|�qS )r �running�userz
*autotracing*r. )�.0�taskr. r. r/ �
<listcomp>� s z<AutoTracer.per_server_limit_recalculated.<locals>.<listcomp>z&Number of running autotracing tasks %s)�lenr\ rX rf Zper_server_limit)r_ Z
running_countr. r. r/ � per_server_limit_recalculated� s z(AutoTracer.per_server_limit_recalculated)� full_listrF c sD |dkr t �� }| j�d|� t�� t| jd� � � fdd�|D �S )zQ
Excludes tasks older than N days from the general list of tasks
NzTask list loaded %s)�daysc s8 g | ]0}|� d �dk r |� d �ndt� �� �kr|�qS )Z
createtimeNr )�getr- � timestamp)rv rw )�
n_days_agor. r/ rx � s z1AutoTracer.excludes_old_tasks.<locals>.<listcomp>)ra r\ rX rf r �nowr Zmin_retracing_interval)r_ r{ r. )r r/ �excludes_old_tasks� s z AutoTracer.excludes_old_tasks)� current_urlrF c C s| t |�}xn| jD ]d}t |d �}|j|jko6|j|jk}t|j|j�oRt|j|j�}|s\|r| j�d||d � dS qW dS )z�
Excludes url from the list if it completely matches the current url
or if domain names match and "*" follows the domain name in the list
rE z2Skipped: URL %s was traced recently. Matched by %sTF)rR r\ rC rD r rX rf )r_ r� �cZ task_data�tZ direct_matchZwildcard_matchr. r. r/ �exclude_thesame_urls� s z AutoTracer.exclude_thesame_urls)�url_total_reqs�domain_total_reqsrF c C s4 | j r0t�t�||��}| j�d|� || jkS dS )z-Check that URL density passes given thresholdzCalculated density %sT)r3 �npZaminZcorrcoefrX rf Zdensity_threshold)r_ r� r� Z url_densityr. r. r/ �pass_by_density� s
zAutoTracer.pass_by_density)�
url_durationsrF c C s0 d| j }t�||�}| j�d||� || jkS )z�
The measure of "slowness" for URL is:
at least N% of its requests take more than X seconds.
N% -- self.slow_duration_percentage
X -- self.slow_duration_threshold
�d z.Calculated %sth percentile %s for min duration)Zslow_duration_percentager� �
percentilerX rf Zslow_duration_threshold)r_ r� Zreversed_percentileZreversed_percentile_valuer. r. r/ �pass_by_slowness_percentile� s
z&AutoTracer.pass_by_slowness_percentilec C s* t �|| j�}| j�d| j|� || jkS )a
The opposite to pass_by_slowness_percentile method.
The measure of "much slowness" for URL is:
at least N% of its requests must take less than X seconds.
N% -- self.max_slow_duration_percentage
X -- self.max_slow_duration_threshold
z.Calculated %sth percentile %s for max duration)r� r� Z max_slow_duration_percentagerX rf Zmax_slow_duration_threshold)r_ r� Zpercentile_valuer. r. r/ � pass_by_max_slowness_percentile s z*AutoTracer.pass_by_max_slowness_percentile)�url_throttled_reqsr� rF c C s8 |dkr dS t |�t |� d }| j�d|� || jkS )zY
Check that percent of throttled requests per URL passes given threshold
NFr� z Calculated throttled percent %s)�sumrX rf Z allowed_throttling_percentage)r_ r� r� Zthrottled_percentr. r. r/ �pass_by_allowed_throttling s
z%AutoTracer.pass_by_allowed_throttling)� wp_statusrF c C s | j rt|�S dS )zx
Check that URLs of a particular domain should be analyzed.
For now we skip non-wordpress sites
T)Zonly_wp�bool)r_ r� r. r. r/ �pass_by_engine s zAutoTracer.pass_by_enginec c s �x
t | j�D �]�}d}| �|j�sZ| j�d� |j}| jj j |7 _ | j j
|7 _
qt t | j|j
��}�x�t| j|j
t|�� ��D �]|\}}|| jkr�q�| j�d|� | j j
d7 _
| �|�d�|d �s�| j�d� | jj jd7 _q�| �|d |j��s&| j�d� | jj jd7 _q�|| }| �|��sZ| j�d � | jj jd7 _q�| �|��s�| j�d
� | jj jd7 _q�|dk�r�| � |j
�}|dk�r�| j�d � | jj j d7 _ q�n|j }t |t t!�"|��t#|d �t#|d �t$|�|j
|d
�V q�W qW dS )a
Select all URLs suitable for auto tracing by very basic rules:
- WP site
- suitable throttling
- suitable density
- measure of "slow" URL
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Nz Skipped by engine: non-wordpresszProcessing URL %sr r� r� z Skipped by throttled percentzSkipped by densityz Skipped by slowness percentilez"Skipped by max slowness percentilez"Skipped by unavailable domain infoZ
url_slow_reqs)r! r# r$ r" r% r& )%r r^ r� Zis_a_wordpress_domainrX rf � urls_numberr] rA r1 r> �dictr rC r r[ �keysZnon_url_fieldsr� r} r2 r� r� r3 r� r4 r� r5 rp r; ru r r- r� Zmeanr� �max) r_ � domain_data� domain_ownerZ
skipped_countZdomain_url_durationsrE �dataZ durationsro r. r. r/ � urls_computationally_filtered& sb
z(AutoTracer.urls_computationally_filtered)�stats_collectedrF c c s� | � � xv| �� D ]j}|j| jkrF| j�d|j� | jj jd7 _q| � |j�rv| j�d|j� | jj j
d7 _
q|V qW dS )z�
From selected by computed thresholds URLs take those for which:
- autotracing enabled
- nginx disabled
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
z'Skipped: autotracing is disabled for %sr z Skipped: nginx is enabled for %sN) �fill_in_disabled_usersr� r& r rX rf r] rA r6 rs r7 )r_ r� rE r. r. r/ r? | s zAutoTracer.urls_selectedc c s� d}t t�}| �� }| j�d|� | �� | _| �� }x�|D ]�}| �|j �r^| j
j j d7 _ q:||k r�||j
| jk r�|d7 }||j
d7 < |V q�| j�d|j | j� | j
j jd7 _q:| j�d|j |� | j
j jd7 _q:W dS )a=
Schedule autotracing by sorted list taking into account the limits:
- no same task for 10 days
- limit per server
- limit per domain
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Return resulting list of URLs scheduled for auto tracing
r zSorted scheduled list %sr z#Skipped URL %s by domain limit (%s)z#Skipped URL %s by server limit (%s)N)r r- � urls_sortedrX rf r� r\ rz r� r r] rA r8 r% Zper_domain_limitr: r9 )r_ Zgeneral_tasks_counterZtasks_counter_per_domainZ sorted_urlsZper_server_smart_limitrE r. r. r/ rh � s.
zAutoTracer.urls_scheduledc C s, t t| �� �dd� dd�}t |dd� dd�S )zd
Sort URLs by total number of requests first
and by average duration second
c S s | j S )N)r! )�ur. r. r/ �<lambda>� � z(AutoTracer.urls_sorted.<locals>.<lambda>T)�key�reversec S s | j S )N)r$ )r� r. r. r/ r� � r� )�sortedr[ r? )r_ Z first_serier. r. r/ r� � s
zAutoTracer.urls_sortedc C s t � | _ dS )z6
Fill internal list of disabled users
N)r )r_ r. r. r/ r� � s z!AutoTracer.fill_in_disabled_usersc
C sp | j dk rly,ttd��}t�t| j �|� W dQ R X W n4 tk
rj } z| j�dt |�� W dd}~X Y nX dS )zI
Save collected statistics for current iteration to file
N�wz*Unable to save iteration stats to file: %s)
r] �openr �json�dumpr �OSErrorrX �warningr+ )r_ �
stats_filerl r. r. r/ rk � s
z AutoTracer.save_iteration_statsc
C sv yFt t��4}t�|�}t|d |d |d tf |d ��}W dQ R X W n& ttjtfk
rl t| j �}Y nX t
|�S )z@
Load statistics for latest iteration from file
r= r> r? rA N) r� r r� �loadr< r0 r� �JSONDecodeError�KeyErrorr= r )r_ r� �_dataZ stat_datar. r. r/ �load_iteration_stats� s
z AutoTracer.load_iteration_statsc C s* | � � }|�tt� d tt� �d�� |S )rH r )r Zdisabled_users_quantity)r� �updater� r ry r )r_ Z stats_loadedr. r. r/ � get_stats� s
zAutoTracer.get_stats)N)N)N)&r' r( r) r* rU rm � staticmethodr� rn r+ r
�objectrp r� rs r- rz r[ r� r� r� r
r� r� r� r� r r r� r? rh r r� r� rk r� r� �
__classcell__r. r. )r` r/ rS m s4
%
W,
rS �__main__zauto_tracer_standalone.log)�filename�level)@r* rZ r� rV rb � collectionsr Z dataclassesr r r r r �fnmatchr �typingr r
r r r
� urllib.parser Znumpyr� Zssa.dbr �commonr Zstorager r r � autotracingr r r �
configurationr Zconfiguration.schemesr �internal.constantsr �internal.utilsr r r re Zxray.shared_libraryZshared_libraryra � ImportErrorrl r r0 r< rB r+ rR rS r' � basicConfig�DEBUGr� r. r. r. r/ �<module>
sP