File: //opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/stat_sender.cpython-37.pyc
B
V�d� �
@ sd d Z dZddlZddlZddlZddlmZ ddlmZ ddl m
Z
m Z ddl m
Z
ddlmZ dd lmZmZ dd
lmZ dd lmZ dd lmZ d
dlmZ d
dlm Z ddl m Z ddl m Z ddl!m"Z"m#Z# ddl$m%Z%m&Z&m'Z'm(Z( G dd� de�Z)e*dk�r`e&� ej+dej,d� ye)� �-� W n4 e k
�r^ Z. ze/e.� e0d
��W ddZ.[.X Y nX dS )zd
This module contains SSA classes for sending e-mails with report
and sending reports to ClickHouse
z ssa.modules� N)� EmailMessage)�partial)�Optional�Any)�get_admin_email)�is_cl_solo_edition)�Session�Response)� HTTPAdapter)�RequestException)�Retry� )�Common)�
DecisionMaker� )� stat_server)�SSAError)�Mailer�render_report_table)� read_sys_id� sentry_init�
duration_cast� format_datec s� e Zd ZdZ� fdd�Zdedd�dd�Zeee d�d d
�Z ee
d �d d
�Z ee
d �dd�Z ee
d�dd�Zeeed�dd��Zeed�dd�Zee
d�dd�Z� ZS ) �StatisticsSenderz4
Send report to ClickHouse and over e-mails
c s� t � �� t�d�| _| j�dt� t� | _dt � d�| _
t � | _ t
dtdg�tddd g�dd
�}t|d �}t� | _| j�d|� t| jjd d
�| j_d S )N� stat_senderz StatisticsSender enabled: %szhttps://z
/api/clos-ssa� �POSTi� i� i� )�total�method_whitelist�status_forcelist�backoff_factor)� max_retries�
)�timeout)�super�__init__�logging� getLogger�logger�info� __package__r �sys_idr � ch_endpointr � mail_senderr � frozensetr
r �session�mountr �request)�selfZ
retry_conf�adapter)� __class__� �H/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/stat_sender.pyr% * s
zStatisticsSender.__init__N)�report�returnc C s0 |dkrt � �� }| jr"| �|� | �|� dS )zi
Send given report over e-mail
If no report given, get it from DecisionMaker API
N)r �get_json_report� summary_notification_enabled� email_report�clickhouse_report)r2 r7 r5 r5 r6 �send; s
zStatisticsSender.send)� report_viewr8 c C s� | � � }|r�|�d�r�t|�\}}| jj|dt|d �|d�}|jt�|�dd|d � d�d� |j|d d|d � d
�d� | j� |� |S d S ) z4
Create and send e-mail with report
�domainsZ
ssa_report�date)Z recipient�templater@ �html�jsonZreport_z.json)�subtype�filenamerB z.htmlN)
�get_mail_recipient�getr r- �_messager �add_attachmentrC �dumpsZ_send)r2 r> Zmail_toZ report_table�mail�msgr5 r5 r6 r; G s
z StatisticsSender.email_report)r8 c
C sH y| � � pt� S tk
rB } z| j�dt|�� W dd}~X Y nX dS )z�
Retrieve a recipient's e-mail:
1. get address from a wmt-api utility
2. if command failed or address is empty, get address of server admin
z get_admin_email failed with: %sN)�wmt_api_report_emailr � Exceptionr( �error�str)r2 �er5 r5 r6 rF ` s z#StatisticsSender.get_mail_recipientc C s� t dd�rdnd}y&tjd|� �dgdddd�j�� }W n> tjtttfk
rv } z| j �
dt |�� d S d }~X Y nX yt �
|��d
��d �S t jk
r� } z| j �
d t |�� W d d }~X Y n" tk
r� | j �
d
|� Y nX d S )zD
Retrieve a recipient's e-mail address from WMT API
T)�skip_jwt_checkz wmt-api-solozwmt-apiz&/usr/share/web-monitoring-tool/wmtbin/z --config-get)�check�text�capture_outputzwmt-api utility failed: %sN�configZ report_emailz!wmt-api returned invalid json: %sz(wmt-api returned unexpected response: %s)r �
subprocess�run�stdout�strip�CalledProcessError�AttributeError�OSError�
ValueErrorr( rO rP rC �loadsrG �JSONDecodeError)r2 �_utilZ api_responserQ r5 r5 r6 rM l s(
z%StatisticsSender.wmt_api_report_emailc
C s� |� d�r�| j�d| j� y"| jj| j| �| �|��d�}W nN tk
r� } z0| jj d|d| jid� t
dt |�� ��|�W dd}~X Y nX | � |�S | j�d � d
S dS ) z+
Send report to ClickHouse
r? zSending POST request to %s)rC zPOST failed with %s�endpoint)�extraz'Failed to POST data to SSA API server: Nz*Report is empty, not sending to ClickHouseF)
rG r( r) r, r/ �post�_ch_pack�clickhouse_formatr rO r rP �_process_response)r2 r> �resprQ r5 r5 r6 r< � s
$
z"StatisticsSender.clickhouse_report)�valuer8 c C s
t | d�S )z<
Pack given value into data field of a dict
)�data)�dict)ri r5 r5 r6 re � s zStatisticsSender._ch_pack)�original_reportr8 c C s` t � }xT|�d�D ]F}|�| j|�d�|�d�|�d�|�d�dd� |�d�D �d �� qW |S )
zd
Format local report for sending to ClickHouse
(required structures differ)
r? �name� slow_urls� slow_reqs�
total_reqsc
S s< g | ]4}|� d �|� d�t|� d��t|� dd��d��qS )rm �reqs_num�average_duration� correlationr )�urlZcount_requests� avg_durationrs )rG r �float)�.0�ur5 r5 r6 �
<listcomp>� s z6StatisticsSender.clickhouse_format.<locals>.<listcomp>�urls)� system_id�domainZcount_slow_urlsZcount_slow_requestsZtotal_requests�details)�listrG �appendr+ )r2 rl Z ch_reportr| r5 r5 r6 rf � s z"StatisticsSender.clickhouse_format)�responser8 c C s� |j s(| jjd|j|jd|jid� dS | j�d|j|j|j� |�� }|d dkrr| jjd|d d |id� dS | j�d
� d S ) z�
Check received response
:param response: a requests.Response object
:return: True in case of success, False otherwise
z&Unable to connect to server with %s:%sZ resp_text)rc Fz [%s:%s] Response received %s�status�okz Received response with status %sr� z Sent to ClickHouse successfullyT) r� r( rO � status_code�reasonrT r) rt rC )r2 r� �resultr5 r5 r6 rg � s z"StatisticsSender._process_response)N)�__name__�
__module__� __qualname__�__doc__r% rk r= r r r; rP rF rM �boolr< � staticmethodr re r~ rf r rg �
__classcell__r5 r5 )r4 r6 r % s r �__main__zstat_sender_standalone.log)rE �level)1r� r* rC r& rW �
email.messager � functoolsr �typingr r �clcommon.cpapir �clcommon.lib.cleditionr �requestsr r Zrequests.adaptersr
Zrequests.exceptionsr Z$requests.packages.urllib3.util.retryr �commonr �decision_makerr �internal.constantsr �internal.exceptionsr Zinternal.mailerr r �internal.utilsr r r r r r� � basicConfig�INFOr= �exc�print�
SystemExitr5 r5 r5 r6 �<module> s< 0