File: //opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/processor.cpython-37.pyc
B
V�dy � @ s� d Z ddlZddlZddlZddlZddlmZmZ ddlmZm Z m
Z
ddl m Z m
Z
ddlmZ ddlmZ dd lmZ dd
lmZ d d lmZmZmZmZ d d
lm Z d dl m Z m Z m Z e G dd� de��Z!dS )z-
This module contains RequestProcessor class
� N)�datetime� timedelta)�Thread�RLock�current_thread)�Callable�Any� )�
AutoTracer)�Common)�
DecisionMaker)�StatisticsSender� )�
session_scope�setup_database�
RequestResult�cleanup_old_data)�SSAError)� singleton� url_split�switch_schedstatsc s� e Zd ZdZdZd#� fdd� Zedd� �Zed�d d
�Z dd �d d
�Z
dd �dd�Z e e
d�dd�Zd$ed�dd�Zdd� Zeeed�dd��Zd d � Zd%d d �Zedd �d!d"�Z� ZS )&�RequestProcessorz^
SSA Request processor implementation.
Only one instance is allowed to be created
�d Nc sx t � �� t�d�| _| j�dt� tdd� |r6|nt� | _ t
� | _ t |d�| _
t� | _t|d�| _| �� g | _d S )NZ
req_processorzProcessor enabled: %sT)�enabled)�engine)�super�__init__�logging� getLogger�logger�info� __package__r r r r �_lockr �decision_makerr
�senderr
� auto_tracer�start_background_routine�_buffer)�selfr )� __class__� �F/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/processor.pyr ) s
zRequestProcessor.__init__c C s
| j d S )zp
Return config file value multiplied by 1000000,
as we receive duration in microseconds
i@B )Zrequests_duration)r( r* r* r+ �configured_duration= s z$RequestProcessor.configured_duration)�reportc
C sJ y| j �|� W n4 tk
rD } z| j�dt|�� W dd}~X Y nX dS )z(
Call Statistics Sender
zStatisticsSender failed: %sN)r$ �sendr r �error�str)r( r- �er* r* r+ �
send_statsE s zRequestProcessor.send_stats)�returnc C s* t | jdd�}|�� | j�d|j� dS )zA
Start dumper|DecisionMaker thread in background
T)�target�daemonz[%s] Routine startedN)r �background_routine�startr r �name)r( �tr* r* r+ r& N s z)RequestProcessor.start_background_routinec C s� x�t �� }|jdkr�|jdkr�| j�dt� j|� | �| j � | j�dt� j|� | �| j
� | j�dt� j|� | �| j �}| j�dt� j|� t | j
� | �| j|� tdd� | �d� q| �|j� qW d S )
z�
Dumps collected stats to file once an hour.
Runs DecisionMaker once a day
Cleanup storage after DecisionMaker run
r z2[%s] Routine thread launching buffer flushing (%s)z-[%s] Routine thread launching AutoTracer (%s)z0[%s] Routine thread launching DecisionMaker (%s)z*[%s] Routine thread launching cleanup (%s)T)r �< N)r �now�minute�hourr r r r8 �
_safe_exec�flush_with_lockr% r# r r r2 r �
_simple_sleep�_sleep_till_next_hour)r( Ztickr- r* r* r+ r6 V s0
z#RequestProcessor.background_routine)�actionr3 c G sR y||� S t k
rL t�� \}}}| jjdt|�||dt�� id� Y nX dS )zACall requested Callable with given args and capture any exceptionz %s failed with exception %s, %sZorig_traceback)�extraN)� Exception�sys�exc_infor � exceptionr0 � traceback�
format_exc)r( rB �args�et�ev�_r* r* r+ r> y s
zRequestProcessor._safe_exec� )�to_sleepc C s" | j �dt� j|� t�|� dS )zP
Log and sleep given number of seconds or 15 minutes by default
z%[%s] Routine thread sleeping for (%s)N)r r r r8 �time�sleep)r( rO r* r* r+ r@ � s z RequestProcessor._simple_sleepc C s* t dd�t |d� �� }| �t|�� dS )zF
Sleep the number of minutes remaining till next hour
r )�hours)�minutesN)r �
total_secondsr@ �int)r( Z start_minuteZ sleep_forr* r* r+ rA � s
z&RequestProcessor._sleep_till_next_hour)� timestampr3 c C s t �| �jS )zN
Takes an hour of a day, to which the given timestamp belongs
)r �
fromtimestampr= )rV r* r* r+ �get_interval_for� s z!RequestProcessor.get_interval_forc C s4 | j � | jd d � }g | _W d Q R X | �|� d S )N)r" r' � flush_buffer)r( �objectsr* r* r+ r? � s z RequestProcessor.flush_with_lockc C s: |dkr| j }|sdS t| j��}|�|� W dQ R X dS )z6
Save in-memory buffer into database.
N)r' r r Zbulk_save_objects)r( rZ �dbr* r* r+ rY � s z RequestProcessor.flush_buffer)�datar3 c C s� |� d�}| �|�r&| j�d|� dS t|�\}}| j�dt� j� g }| j�n | j� t
|||d |d |d | j k|d |d |d |d
d � � t | j�| j
kr�| jdd� }g | _W dQ R X | �|� | j�d t� j� dS )
z,
Process given request data
�urlz
%s ignoredNz-[%s] Acquires lock to handle request countersrV �duration�hitting_limits�throttled_time�io_throttled_time� wordpress) �domain�pathrV r^ Zis_slow_requestr_ r` ra rb z-[%s] Released lock to handle request counters)�getZ
is_ignoredr �debugr r r8 r" r' �appendr r, �len� BUFFER_SIZErY )r( r\ r] rc �uriZobjects_per_threadr* r* r+ �handle� s4
zRequestProcessor.handle)N)rN )N)�__name__�
__module__� __qualname__�__doc__ri r �propertyr, �dictr2 r& r6 r r r> rU r@ rA � staticmethodrX r? rY rk �
__classcell__r* r* )r) r+ r s #
r )"ro r rE rP rH r r � threadingr r r �typingr r Z
autotracerr
�commonr r# r Z stat_senderr
r[ r r r r �internal.exceptionsr �internal.utilsr r r r r* r* r* r+ �<module>
s