File: //usr/lib/python3.6/site-packages/S3/__pycache__/MultiPart.cpython-36.opt-1.pyc
3
�9peQ5 � @ s� d dl mZ d dlZd dlmZmZmZmZ ddlm Z ddl
m Z ddl m
Z
ddlmZmZmZmZ dd lmZ d
ZG d d � d e�ZdS )� )�absolute_importN)�debug�info�warning�error� )�calculateChecksum)�ParameterError)�S3UriS3)�getTextFromXml�getTreeFromXml�s3_quote�
parseNodes)�
formatSizei c @ sl e Zd ZdZdZdZdZddd�Zdd� Zd d
� Z d d � Z
d dd�Z d dd�Z d dd�Z
dd� Zdd� ZdS ) �MultiPartUploadz<Supports MultiPartUpload and MultiPartUpload(Copy) operation� i Nc C s� || _ d | _d | _|| _|| _i | _|p*i | _t|t�rZ|| _|sJt d��| j j
j t }n|| _| j j
j
t }|| _| j� | _d S )Nz8Source size is missing for MultipartUploadCopy operation)�s3� file_stream�src_uri�src_size�dst_uri�parts�headers_baseline�
isinstancer
r �configZ multipart_copy_chunk_size_mb�SIZE_1MBZmultipart_chunk_size_mb�
chunk_size�initiate_multipart_upload� upload_id)�selfr �srcr r r Zc_size� r! � /usr/lib/python3.6/MultiPart.py�__init__ s
zMultiPartUpload.__init__c C s^ | j j||�}t� }xD|D ]<}y"|d |d d�|t|d �< W q tk
rT Y qX qW |S )N�ETagZSize)�checksum�sizeZ
PartNumber)r Zlist_multipart�dict�int�KeyError)r �urir Z part_listr �elemr! r! r" �get_parts_information3 s
z%MultiPartUpload.get_parts_informationc C s� d}| j j|�}xv|D ]n}yT|d }|d }td||j� f � ||j� krl|rhtd|tjd |f ��|}W q tk
r� Y qX qW |S )N� �UploadIdZKeyzmp_path: %s, object: %sz�More than one UploadId for URI %s. Disable multipart upload, or use
%s multipart %s
to list the Ids, then pass a unique --upload-id into the put command.r )r Z
get_multipartr �object�
ValueError�sys�argvr) )r r* r Zmultipart_listZmpuploadZ mp_upload_idZmp_pathr! r! r" �get_unique_upload_idB s
z$MultiPartUpload.get_unique_upload_idc C s� | j jjr| j jj| _n | j jjr2| j| j�| _nd| _| jsz| j jd| j| jddid�}| j j|�}|d }t |d�| _| jS )z�
Begin a multipart upload
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadInitiate.html
r- � OBJECT_POSTZuploadsN)r* �headers�
uri_params�datar. )
r r r � put_continuer3 r �create_requestr � send_requestr )r �request�responser7 r! r! r" r W s
z)MultiPartUpload.initiate_multipart_uploadr- c C sZ | j std��i }| jr$| jj� }n| jj}| jjjrF| j | j
| j �}|rRd| }|| j
j� d�}d}| j �r�| j }| j | j | j | j o�d }t
d||f � x�|dk�r|| j |d }t| j | | j �} || 8 }d||dt| d d
� |f |d < yB| j�r| j||| ||j|�d � n| j||| ||j|�d � W n: td
||tjd | j
| j tjd | j f � � Y nX |d7 }q�W t
d|d � dS t
d| � x�| jj| j �}
d}t|
�} d|dt| d d
� |f |d < |
�s�P y | j||| ||
|j|�d � W n: td||tjd | j
| j tjd | j f � � Y nX |d7 }�q�W t
d|d � dS )z�
Execute a full multipart upload on a file
Returns the seq/etag dict
TODO use num_processes to thread it
zAAttempting to use a multipart upload that has not been initiated.� )�sourceZ destinationr z#MultiPart: Uploading %s in %d partsr z[part %d of %d, %s]%sz%d%sBT)Zhuman_readableZextra)�
remote_statusz�
Upload of '%s' part %d failed. Use
%s abortmp %s %s
to abort the upload, or
%s --upload-id %s put ...
to continue the upload.z$MultiPart: Upload finished: %d partsNz MultiPart: Uploading from %sz[part %d of -, %s]%szw
Upload of '%s' part %d failed. Use
%s abortmp %s %s
to abort, or
%s --upload-id %s put ...
to continue the upload.)r r r r* r Z stream_namer r r8 r, r r r r �minr � upload_part�get� copy_partr r1 r2 �read�len) r Z extra_labelZremote_statuses�filename�labels�seqZ size_leftZnr_parts�offsetZcurrent_chunk_size�bufferr! r! r" �upload_all_partsm s�
z MultiPartUpload.upload_all_partsc
C s t d|| j|f � |dk r�t|d �|kr�t|| j||| jjj�}|d jd�}||kr~t d| j
|f � |d | j |< dS t d||| j
|f � n t dt|d �|| j
|f � d t |�i} d
| | jd �}
| jj
d | j
| |
d
�} | jj| | j||||d�} | d jdd�jd�| j |< | S )z�
Upload a file chunk
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadUploadPart.html
z"Uploading part %i of %r (%s bytes)Nr& r% z"'z:MultiPart: size and md5sum match for %s part %d, skipping.zJMultiPart: checksum (%s vs %s) does not match for %s part %d, reuploading.zFMultiPart: size (%d vs %d) does not match for %s part %d, reuploading.zcontent-lengthz%s)�
partNumber�uploadId�
OBJECT_PUT)r* r5 r6 )rI r r5 �etagr- )r r r( r r r r Z
send_chunk�stripr r r �strr9 Z send_filerB )
r rH rI r rG rJ r? r% Zremote_checksumr5 �query_string_paramsr; r< r! r! r" rA � s8
zMultiPartUpload.upload_partc
C s� t d|| j|f � dtd| jj� | jj� f ddd�i}d||| d f |d < d
| | jd �}| jjd | j||d
�}d|d< | jj |||�} t
| d d�p�d| j |< | S )z�
Copy a remote file chunk
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadUploadPart.html
http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
z Copying part %i of %r (%s bytes)zx-amz-copy-sourcez/%s/%sFT)Zquote_backslashesZunicode_outputz bytes=%d-%dr zx-amz-copy-source-rangez%s)rL rM rN )r* r5 r6 z remote copy�actionr7 r$ r- ) r r r
r Zbucketr/ r r9 r Zsend_request_with_progressr r )
r rH rI r rG r? r5 rR r; r< r! r! r" rC � s"
zMultiPartUpload.copy_partc C s� t d| j � g }d}x(| jj� D ]\}}|j|||f � q"W ddj|� }dtt|��i}| jj d| j
||d| jid�}| jj |�}|S ) z�
Finish a multipart upload
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadComplete.html
z MultiPart: Completing upload: %sz7<Part><PartNumber>%i</PartNumber><ETag>%s</ETag></Part>z5<CompleteMultipartUpload>%s</CompleteMultipartUpload>r- zcontent-lengthr4 rM )r* r5 �bodyr6 ) r r r �items�append�joinrQ rE r r9 r r: ) r Z parts_xmlZpart_xmlrH rO rT r5 r; r< r! r! r" �complete_multipart_upload$ s
z)MultiPartUpload.complete_multipart_uploadc C s t d| j � d}|S )z�
Abort multipart upload
http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadAbort.html
z MultiPart: Aborting upload: %sN)r r )r r< r! r! r" � abort_upload: s z MultiPartUpload.abort_uploadi i i P )NN)r- )r- N)N)�__name__�
__module__� __qualname__�__doc__ZMIN_CHUNK_SIZE_MBZMAX_CHUNK_SIZE_MBZ
MAX_FILE_SIZEr# r, r3 r rK rA rC rX rY r! r! r! r" r s
`
(
.r i )Z
__future__r r1 Zloggingr r r r ZCryptor Z
Exceptionsr ZS3Urir
Z BaseUtilsr r r
r ZUtilsr r r/ r r! r! r! r" �<module> s