File: //usr/local/lib/python3.10/site-packages/pip/_internal/index/__pycache__/collector.cpython-310.pyc
o
��ix@ � @ s� d Z ddlZddlZddlZddlZddlZddlZddlZddl Z
ddl Z
ddl m
Z
ddlmZ ddlmZmZmZmZmZmZmZmZmZmZmZ ddl m Z ddl m Z ddl m!Z!m"Z" dd l#m$Z$ dd
l%m&Z& dd l'm(Z( dd l)m*Z* dd
l+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4m5Z5m6Z6 er�ddlm7Z7 ne8Z7e�9e:�Z;ee<e<f Z=de<dee< fdd�Z>G dd� de?�Z@de ddfdd �ZAG d d � d e?�ZBde<d e*ddfd d!�ZCde<d e*de fd"d#�ZDd$e=dee< fd%d&�ZEG d'd(� d(�ZFG d)d*� d*e7�ZGd+eGdeGfd,d-�ZHeHd.d/dee& fd0d1��ZIG d2d/� d/�ZJG d3d4� d4e
�ZK dEd5e&d6ee<e?f d7eed8 ddfd9d:�ZL ;dFde d<eMdeJfd=d>�ZNd5e&d e*ded/ fd?d@�ZOG dAdB� dBe�ZPG dCdD� dD�ZQdS )GzO
The main purpose of this module is to expose LinkCollector.collect_sources().
� N)�
HTMLParser)�Values) �
TYPE_CHECKING�Callable�Dict�Iterable�List�MutableMapping�
NamedTuple�Optional�Sequence�Tuple�Union)�requests)�Response)�
RetryError�SSLError)�NetworkConnectionError)�Link)� SearchScope)�
PipSession)�raise_for_status)�is_archive_file��redact_auth_from_url)�vcs� )�CandidatesFromPage�
LinkSource� build_source)�Protocol�url�returnc C s6 t jD ]}| �� �|�r| t|� dv r| S qdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r �schemes�lower�
startswith�len)r! �scheme� r( �H/usr/local/lib/python3.10/site-packages/pip/_internal/index/collector.py�_match_vcs_scheme7 s
�r* c s* e Zd Zdededdf� fdd� Z� ZS )�_NotAPIContent� content_type� request_descr" Nc s t � �||� || _|| _d S �N)�super�__init__r, r- )�selfr, r- �� __class__r( r) r0 C s
z_NotAPIContent.__init__)�__name__�
__module__� __qualname__�strr0 �
__classcell__r( r( r2 r) r+ B s "r+ �responsec C s2 | j �dd�}|�� }|�d�rdS t|| jj��)z�
Check the Content-Type header to ensure the response contains a Simple
API Response.
Raises `_NotAPIContent` if the content type is not a valid content-type.
� Content-Type�Unknown)z text/htmlz#application/vnd.pypi.simple.v1+html�#application/vnd.pypi.simple.v1+jsonN)�headers�getr$ r% r+ �request�method)r9 r, �content_type_lr( r( r) �_ensure_api_headerI s �rB c @ s e Zd ZdS )�_NotHTTPN)r4 r5 r6 r( r( r( r) rC _ s rC �sessionc C sF t j�| �\}}}}}|dvrt� �|j| dd�}t|� t|� dS )z�
Send a HEAD request to the URL, and ensure the response contains a simple
API Response.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotAPIContent` if the content type is not a valid content type.
> �https�httpT)�allow_redirectsN)�urllib�parse�urlsplitrC �headr rB )r! rD r' �netloc�path�query�fragment�respr( r( r) �_ensure_api_responsec s rQ c C sx t t| �j�r
t| |d� t�dt| �� |j| d�g d��dd�d�}t |� t
|� t�dt| �|j �d d
�� |S ) aY Access an Simple API response with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML or Simple API, to avoid downloading a
large file. Raise `_NotHTTP` if the content type cannot be determined, or
`_NotAPIContent` if it is not HTML or a Simple API.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got a Simple API response,
and raise `_NotAPIContent` otherwise.
�rD zGetting page %sz, )r< z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)�Acceptz
Cache-Control)r= zFetched page %s as %sr: r; ) r r �filenamerQ �logger�debugr r> �joinr rB r= )r! rD rP r( r( r) �_get_simple_responseu s&
��� �rX r= c C s< | r d| v r t j�� }| d |d< |�d�}|r t|�S dS )z=Determine if we have any encoding information in our headers.r: z content-type�charsetN)�email�message�Message� get_paramr7 )r= �mrY r( r( r) �_get_encoding_from_headers� s
r_ c @ s6 e Zd Zd dd�Zdedefdd �Zdefd
d �ZdS )
�CacheablePageContent�page� IndexContentr" Nc C s |j sJ �|| _d S r. )�cache_link_parsingra �r1 ra r( r( r) r0 � s
z CacheablePageContent.__init__�otherc C s t |t| ��o| jj|jjkS r. )�
isinstance�typera r! )r1 re r( r( r) �__eq__� s zCacheablePageContent.__eq__c C s t | jj�S r. )�hashra r! �r1 r( r( r) �__hash__� s z CacheablePageContent.__hash__)ra rb r" N) r4 r5 r6 r0 �object�boolrh �intrk r( r( r( r) r` � s
r` c @ s" e Zd Zdddee fdd�ZdS )�
ParseLinksra rb r" c C s d S r. r( rd r( r( r) �__call__� s zParseLinks.__call__N)r4 r5 r6 r r rp r( r( r( r) ro � s ro �fnc sP t jdd�dtdtt f� fdd� ��t �� �dddtt f� �fd d
� �}|S ) z�
Given a function that parses an Iterable[Link] from an IndexContent, cache the
function's result (keyed by CacheablePageContent), unless the IndexContent
`page` has `page.cache_link_parsing == False`.
N)�maxsize�cacheable_pager" c s t � | j��S r. )�listra )rs )rq r( r) �wrapper� s z*with_cached_index_content.<locals>.wrapperra rb c s | j r �t| ��S t� | ��S r. )rc r` rt )ra �rq ru r( r) �wrapper_wrapper� s z2with_cached_index_content.<locals>.wrapper_wrapper)� functools� lru_cacher` r r �wraps)rq rw r( rv r) �with_cached_index_content� s
r{ ra rb c
c s� � | j �� }|�d�r+t�| j�}|�dg �D ]}t�|| j �}|du r%q|V qdS t
| j �}| j p4d}|� | j�
|�� | j }|jpE|}|jD ]} tj| ||d�}|du rXqI|V qIdS )z\
Parse a Simple API's Index Content, and yield its anchor elements as Link objects.
r< �filesNzutf-8)�page_url�base_url)r, r$ r% �json�loads�contentr> r � from_jsonr! �HTMLLinkParser�encoding�feed�decoder~ �anchors� from_element)
ra rA �data�file�link�parserr� r! r~ �anchorr( r( r) � parse_links� s* �
�r� c
@ sH e Zd ZdZ ddededee dededd f d
d �Zdefd d
�Z d S )rb z5Represents one response (or page), along with its URLTr� r, r� r! rc r" Nc C s" || _ || _|| _|| _|| _dS )am
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
N)r� r, r� r! rc )r1 r� r, r� r! rc r( r( r) r0 s
zIndexContent.__init__c C s
t | j�S r. )r r! rj r( r( r) �__str__ s
zIndexContent.__str__�T)
r4 r5 r6 �__doc__�bytesr7 r rm r0 r� r( r( r( r) rb s" ������
�c sv e Zd ZdZdeddf� fdd� Zdedeeeee f ddfd d
�Z deeeee f dee fd d �Z
� Z S )
r� zf
HTMLParser that keeps the first base HREF and a list of all anchor
elements' attributes.
r! r" Nc s$ t � jdd� || _d | _g | _d S )NT)�convert_charrefs)r/ r0 r! r~ r� )r1 r! r2 r( r) r0 # s
zHTMLLinkParser.__init__�tag�attrsc C sR |dkr| j d u r| �|�}|d ur|| _ d S d S |dkr'| j�t|�� d S d S )N�base�a)r~ �get_hrefr� �append�dict)r1 r� r� �hrefr( r( r) �handle_starttag* s
��z HTMLLinkParser.handle_starttagc C s" |D ] \}}|dkr| S qd S )Nr� r( )r1 r� �name�valuer( r( r) r� 2 s
�zHTMLLinkParser.get_href) r4 r5 r6 r� r7 r0 r r
r r� r� r8 r( r( r2 r) r� s
&.r� r� �reason�meth).Nc C s |d u rt j}|d| |� d S )Nz%Could not fetch URL %s: %s - skipping)rU rV )r� r� r� r( r( r) �_handle_get_simple_fail9 s r� Trc c C s& t | j�}t| j| jd || j|d�S )Nr: )r� r! rc )r_ r= rb r� r! )r9 rc r� r( r( r) �_make_index_contentC s
�r� c
C s� | j �dd�d }t|�}|rt�d|| � d S tj�|�\}}}}}}|dkrHtj �
tj � |��rH|�
d�s;|d7 }tj�|d�}t�d|� zt||d �}W n� ty` t�d
| � Y d S ty| } zt�d | |j|j� W Y d }~d S d }~w ty� } z t| |� W Y d }~d S d }~w ty� } z t| |� W Y d }~d S d }~w ty� } zd } | t|�7 } t| | tjd
� W Y d }~d S d }~w tj y� } zt| d|� �� W Y d }~d S d }~w tj y� t| d� Y d S w t || j d�S )N�#r r zICannot look at %s URL %s because it does not support lookup as web pages.r� �/z
index.htmlz# file: URL is directory, getting %srR z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.z�Skipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r� zconnection error: z timed out)rc ) r! �splitr* rU �warningrH rI �urlparse�osrM �isdirr? � url2pathname�endswith�urljoinrV rX rC r+ r- r, r r� r r r7 �infor �ConnectionError�Timeoutr� rc )
r� rD r! �
vcs_schemer' �_rM rP �excr� r( r( r) �_get_index_contentP sn �
� ����
�� �� ���� �r� c @ s. e Zd ZU eee ed< eee ed< dS )�CollectedSources�
find_links�
index_urlsN)r4 r5 r6 r r r �__annotations__r( r( r( r) r� � s
r� c
@ s� e Zd ZdZdededdfdd�Ze dded ed
e dd fd d ��Z
e de e
fd
d��Zdedee fdd�Zde
dedefdd�ZdS )�
LinkCollectorz�
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_sources() method.
rD � search_scoper" Nc C s || _ || _d S r. )r� rD )r1 rD r� r( r( r) r0 � s
zLinkCollector.__init__F�options�suppress_no_indexc C sd |j g|j }|jr |s t�dd�dd� |D ��� g }|jp g }tj|||jd�}t ||d�}|S )z�
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
zIgnoring indexes: %s�,c s s � | ]}t |�V qd S r. r )�.0r! r( r( r) � <genexpr>� s � z'LinkCollector.create.<locals>.<genexpr>)r� r� �no_index)rD r� )
� index_url�extra_index_urlsr� rU rV rW r� r �creater� )�clsrD r� r� r� r� r� �link_collectorr( r( r) r� � s$
�
��zLinkCollector.createc C s | j jS r. )r� r� rj r( r( r) r� � s zLinkCollector.find_links�locationc C s t || jd�S )z>
Fetch an HTML page containing package links.
rR )r� rD )r1 r� r( r( r) �fetch_response� s z LinkCollector.fetch_response� project_name�candidates_from_pagec s� t �� �fdd��j�|�D ���� }t �� �fdd��jD ���� }t�tj �rIdd� t
� ||�D �}t |�� d|� d�g| }t�
d�|�� tt|�t|�d �S )
Nc 3 �& � | ]}t |� �jjd d d�V qdS )F�r� �page_validator�
expand_dirrc N�r rD �is_secure_origin�r� �loc�r� r1 r( r) r� � � � ��
�z0LinkCollector.collect_sources.<locals>.<genexpr>c 3 r� )Tr� Nr� r� r� r( r) r� � r� c S s* g | ]}|d ur|j d urd|j � ��qS )Nz* )r� )r� �sr( r( r) �
<listcomp>� s
�z1LinkCollector.collect_sources.<locals>.<listcomp>z' location(s) to search for versions of �:�
)r� r� )� collections� OrderedDictr� �get_index_urls_locations�valuesr� rU � isEnabledFor�logging�DEBUG� itertools�chainr& rV rW r� rt )r1 r� r� �index_url_sources�find_links_sources�linesr( r� r) �collect_sources� s2
� �
� �
�
����z LinkCollector.collect_sources)F)r4 r5 r6 r� r r r0 � classmethodr rm r� �propertyr r7 r� r r rb r� r r� r� r( r( r( r) r� � s< ��
����� �!���r� r. r� )Rr� r� �
email.messagerZ rx r� r r� r� � urllib.parserH �urllib.request� html.parserr �optparser �typingr r r r r r r
r r r
r � pip._vendorr Zpip._vendor.requestsr Z pip._vendor.requests.exceptionsr r �pip._internal.exceptionsr �pip._internal.models.linkr �!pip._internal.models.search_scoper � pip._internal.network.sessionr �pip._internal.network.utilsr � pip._internal.utils.filetypesr �pip._internal.utils.miscr �pip._internal.vcsr �sourcesr r r r rl � getLoggerr4 rU r7 �ResponseHeadersr* � Exceptionr+ rB rC rQ rX r_ r` ro r{ r� rb r� r� rm r� r� r� r� r( r( r( r) �<module> s� 4
? ��
�
�
� ���
�
=