| Server IP : 127.0.0.1 / Your IP : 216.73.216.109 Web Server : Apache/2.4.54 (Win64) OpenSSL/1.1.1q PHP/8.1.10 System : Windows NT DESKTOP-E5T4RUN 10.0 build 19045 (Windows 10) AMD64 User : SERVERWEB ( 0) PHP Version : 8.1.10 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : OFF | Perl : OFF | Python : OFF | Sudo : OFF | Pkexec : OFF Directory : C:/cygwin64/lib/python3.9/site-packages/pip/_internal/index/__pycache__/ |
Upload File : |
a
W��e�@ � @ s� d Z ddlZddlZddlZddlZddlZddlZddlZddl Z
ddl Z
ddl m
Z
ddlmZ ddlmZmZmZmZmZmZmZmZmZmZmZ ddl m Z ddl m Z ddl m!Z!m"Z" dd l#m$Z$ dd
l%m&Z& dd l'm(Z( dd l)m*Z* dd
l+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4m5Z5m6Z6 e�rHddlm7Z7 ne8Z7e�9e:�Z;ee<e<f Z=e<ee< d�dd�Z>G dd� de?�Z@e dd�dd�ZAG d d � d e?�ZBe<e*dd �d d �ZCe<e*e d �d!d"�ZDe=ee< d#�d$d%�ZEG d&d'� d'�ZFG d(d)� d)e7�ZGeGeGd*�d+d,�ZHeHd-ee& d.�d/d0��ZIG d1d-� d-�ZJG d2d3� d3e
�ZKdCe&ee<e?f eed4 dd5�d6d7�ZLdDe eMeJd9�d:d;�ZNe&e*ed- d<�d=d>�ZOG d?d@� d@e�ZPG dAdB� dB�ZQdS )EzO
The main purpose of this module is to expose LinkCollector.collect_sources().
� N)�
HTMLParser)�Values) �
TYPE_CHECKING�Callable�Dict�Iterable�List�MutableMapping�
NamedTuple�Optional�Sequence�Tuple�Union)�requests)�Response)�
RetryError�SSLError)�NetworkConnectionError)�Link)� SearchScope)�
PipSession)�raise_for_status)�is_archive_file��redact_auth_from_url)�vcs� )�CandidatesFromPage�
LinkSource� build_source)�Protocol��url�returnc C s6 t jD ]*}| �� �|�r| t|� dv r| S qdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r Zschemes�lower�
startswith�len)r" �scheme� r( �A/usr/lib/python3.9/site-packages/pip/_internal/index/collector.py�_match_vcs_scheme7 s
r* c s& e Zd Zeedd�� fdd� Z� ZS )�_NotAPIContentN)� content_type� request_descr# c s t � �||� || _|| _d S �N)�super�__init__r, r- )�selfr, r- �� __class__r( r) r0 C s z_NotAPIContent.__init__)�__name__�
__module__� __qualname__�strr0 �
__classcell__r( r( r2 r) r+ B s r+ )�responser# c C s6 | j �dd�}|�� }|�d�r$dS t|| jj��dS )z�
Check the Content-Type header to ensure the response contains a Simple
API Response.
Raises `_NotAPIContent` if the content type is not a valid content-type.
� Content-Type�Unknown)z text/htmlz#application/vnd.pypi.simple.v1+html�#application/vnd.pypi.simple.v1+jsonN)�headers�getr$ r% r+ �request�method)r9 r, �content_type_lr( r( r) �_ensure_api_headerI s �rB c @ s e Zd ZdS )�_NotHTTPN)r4 r5 r6 r( r( r( r) rC _ s rC )r" �sessionr# c C sF t j�| �\}}}}}|dvr$t� �|j| dd�}t|� t|� dS )z�
Send a HEAD request to the URL, and ensure the response contains a simple
API Response.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotAPIContent` if the content type is not a valid content type.
> �http�httpsT)Zallow_redirectsN)�urllib�parse�urlsplitrC �headr rB )r" rD r' �netloc�path�query�fragment�respr( r( r) �_ensure_api_responsec s rP c C sx t t| �j�rt| |d� t�dt| �� |j| d�g d��dd�d�}t |� t
|� t�dt| �|j �d d
�� |S ) aY Access an Simple API response with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML or Simple API, to avoid downloading a
large file. Raise `_NotHTTP` if the content type cannot be determined, or
`_NotAPIContent` if it is not HTML or a Simple API.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got a Simple API response,
and raise `_NotAPIContent` otherwise.
�rD zGetting page %sz, )r< z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)ZAcceptz
Cache-Control)r= zFetched page %s as %sr: r; ) r r �filenamerP �logger�debugr r> �joinr rB r= )r" rD rO r( r( r) �_get_simple_responseu s&
��� �rV )r= r# c C s< | r8d| v r8t j�� }| d |d< |�d�}|r8t|�S dS )z=Determine if we have any encoding information in our headers.r: z content-type�charsetN)�email�messageZMessageZ get_paramr7 )r= �mrW r( r( r) �_get_encoding_from_headers� s
r[ c @ s: e Zd Zddd�dd�Zeed�dd�Zed �d
d �ZdS ) �CacheablePageContent� IndexContentN��pager# c C s |j s
J �|| _d S r. )�cache_link_parsingr_ �r1 r_ r( r( r) r0 � s
z CacheablePageContent.__init__)�otherr# c C s t |t| ��o | jj|jjkS r. )�
isinstance�typer_ r" )r1 rb r( r( r) �__eq__� s zCacheablePageContent.__eq__�r# c C s t | jj�S r. )�hashr_ r" �r1 r( r( r) �__hash__� s z CacheablePageContent.__hash__) r4 r5 r6 r0 �object�boolre �intri r( r( r( r) r\ � s r\ c @ s e Zd Zdee d�dd�ZdS )�
ParseLinksr] r^ c C s d S r. r( ra r( r( r) �__call__� s zParseLinks.__call__N)r4 r5 r6 r r rn r( r( r( r) rm � s rm )�fnr# c sL t jdd�ttt d�� fdd� ��t �� �dtt d�� �fdd � �}|S )
z�
Given a function that parses an Iterable[Link] from an IndexContent, cache the
function's result (keyed by CacheablePageContent), unless the IndexContent
`page` has `page.cache_link_parsing == False`.
N)�maxsize)�cacheable_pager# c s t � | j��S r. )�listr_ )rq )ro r( r) �wrapper� s z*with_cached_index_content.<locals>.wrapperr] r^ c s | j r�t| ��S t� | ��S r. )r` r\ rr )r_ �ro rs r( r) �wrapper_wrapper� s z2with_cached_index_content.<locals>.wrapper_wrapper)� functools� lru_cacher\ r r �wraps)ro ru r( rt r) �with_cached_index_content� s
ry r] r^ c
c s� | j �� }|�d�rTt�| j�}|�dg �D ]"}t�|| j �}|du rHq,|V q,dS t
| j �}| j pfd}|� | j�
|�� | j }|jp�|}|jD ]$} tj| ||d�}|du r�q�|V q�dS )z\
Parse a Simple API's Index Content, and yield its anchor elements as Link objects.
r< �filesNzutf-8)Zpage_url�base_url)r, r$ r% �json�loads�contentr> r Z from_jsonr" �HTMLLinkParser�encodingZfeed�decoder{ �anchorsZ from_element)
r_ rA �data�file�link�parserr� r" r{ �anchorr( r( r) � parse_links� s&
r� c @ s< e Zd ZdZd
eeee eedd�dd�Zed�dd �Z dS ) r] z5Represents one response (or page), along with its URLTN)r~ r, r� r" r` r# c C s" || _ || _|| _|| _|| _dS )am
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
N)r~ r, r� r" r` )r1 r~ r, r� r" r` r( r( r) r0 s
zIndexContent.__init__rf c C s
t | j�S r. )r r" rh r( r( r) �__str__ s zIndexContent.__str__)T)
r4 r5 r6 �__doc__�bytesr7 r rk r0 r� r( r( r( r) r] s �� c sn e Zd ZdZedd�� fdd� Zeeeeee f dd�dd�Z eeeee f ee d �d
d �Z
� Z S ) r zf
HTMLParser that keeps the first base HREF and a list of all anchor
elements' attributes.
Nr! c s$ t � jdd� || _d | _g | _d S )NT)Zconvert_charrefs)r/ r0 r" r{ r� )r1 r" r2 r( r) r0 # s zHTMLLinkParser.__init__)�tag�attrsr# c C sH |dkr,| j d u r,| �|�}|d urD|| _ n|dkrD| j�t|�� d S )N�base�a)r{ �get_hrefr� �append�dict)r1 r� r� �hrefr( r( r) �handle_starttag* s
z HTMLLinkParser.handle_starttag)r� r# c C s" |D ]\}}|dkr| S qd S )Nr� r( )r1 r� �name�valuer( r( r) r� 2 s
zHTMLLinkParser.get_href) r4 r5 r6 r� r7 r0 r r
r r� r� r8 r( r( r2 r) r s "r ).N)r� �reason�methr# c C s |d u rt j}|d| |� d S )Nz%Could not fetch URL %s: %s - skipping)rS rT )r� r� r� r( r( r) �_handle_get_simple_fail9 s r� T)r9 r` r# c C s& t | j�}t| j| jd || j|d�S )Nr: )r� r" r` )r[ r= r] r~ r" )r9 r` r� r( r( r) �_make_index_contentC s
�r� )r� rD r# c
C s | j �dd�d }t|�}|r0t�d|| � d S tj�|�\}}}}}}|dkr�tj �
tj � |��r�|�
d�sv|d7 }tj�|d�}t�d|� zt||d �}W �nL ty� t�d
| � Y �n< ty� } z"t�d | |j|j� W Y d }~�n d }~0 t�y( } zt| |� W Y d }~n�d }~0 t�yV } zt| |� W Y d }~n�d }~0 t�y� } z,d } | t|�7 } t| | tjd
� W Y d }~nld }~0 tj �y� } z t| d|� �� W Y d }~n6d }~0 tj �y� t| d� Y n0 t || j d�S d S )N�#r r zICannot look at %s URL %s because it does not support lookup as web pages.r� �/z
index.htmlz# file: URL is directory, getting %srQ z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.z�Skipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r� zconnection error: z timed out)r` ) r" �splitr* rS ZwarningrG rH �urlparse�osrL �isdirr? Z url2pathname�endswith�urljoinrT rV rC r+ r- r, r r� r r r7 �infor �ConnectionErrorZTimeoutr� r` )
r� rD r" Z
vcs_schemer' �_rL rO �excr� r( r( r) �_get_index_contentP sV �
�
� $$r� c @ s. e Zd ZU eee ed< eee ed< dS )�CollectedSources�
find_links�
index_urlsN)r4 r5 r6 r r r �__annotations__r( r( r( r) r� � s
r� c @ sx e Zd ZdZeedd�dd�Zedeee d d�dd ��Z
e e e
d
�d d ��Zeee d
�dd�Ze
eed�dd�ZdS )�
LinkCollectorz�
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_sources() method.
N)rD � search_scoper# c C s || _ || _d S r. )r� rD )r1 rD r� r( r( r) r0 � s zLinkCollector.__init__F)rD �options�suppress_no_indexr# c C sd |j g|j }|jr8|s8t�dd�dd� |D ��� g }|jp@g }tj|||jd�}t ||d�}|S )z�
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
zIgnoring indexes: %s�,c s s | ]}t |�V qd S r. r )�.0r" r( r( r) � <genexpr>� � z'LinkCollector.create.<locals>.<genexpr>)r� r� �no_index)rD r� )
Z index_urlZextra_index_urlsr� rS rT rU r� r �creater� )�clsrD r� r� r� r� r� Zlink_collectorr( r( r) r� � s$
�
��zLinkCollector.createrf c C s | j jS r. )r� r� rh r( r( r) r� � s zLinkCollector.find_links)�locationr# c C s t || jd�S )z>
Fetch an HTML page containing package links.
rQ )r� rD )r1 r� r( r( r) �fetch_response� s z LinkCollector.fetch_response)� project_name�candidates_from_pager# c s� t �� ��fdd��j���D ���� }t �� ��fdd��jD ���� }t�tj �r�dd� t
� ||�D �}t |�� d�� d�g| }t�
d�|�� tt|�t|�d �S )
Nc 3 s& | ] }t |� �jjd d �d�V qdS )F�r� Zpage_validatorZ
expand_dirr` r� N�r rD Zis_secure_origin�r� �loc�r� r� r1 r( r) r� � s ��z0LinkCollector.collect_sources.<locals>.<genexpr>c 3 s& | ] }t |� �jjd d �d�V qdS )Tr� Nr� r� r� r( r) r� � s ��c S s* g | ]"}|d ur|j d urd|j � ��qS )Nz* )r� )r� �sr( r( r) �
<listcomp>� s �z1LinkCollector.collect_sources.<locals>.<listcomp>z' location(s) to search for versions of �:�
)r� r� )� collections� OrderedDictr� Zget_index_urls_locations�valuesr� rS Z isEnabledFor�logging�DEBUG� itertools�chainr& rT rU r� rr )r1 r� r� Zindex_url_sourcesZfind_links_sources�linesr( r� r) �collect_sources� s*
� �
�
����z LinkCollector.collect_sources)F)r4 r5 r6 r� r r r0 � classmethodr rk r� �propertyr r7 r� r r r] r� r r� r� r( r( r( r) r� � s( � ��!�r� )N)T)Rr� r� Z
email.messagerX rv r� r| r� r� � urllib.parserG Zurllib.requestZ html.parserr Zoptparser �typingr r r r r r r
r r r
r Z pip._vendorr Zpip._vendor.requestsr Z pip._vendor.requests.exceptionsr r Zpip._internal.exceptionsr Zpip._internal.models.linkr Z!pip._internal.models.search_scoper Z pip._internal.network.sessionr Zpip._internal.network.utilsr Z pip._internal.utils.filetypesr Zpip._internal.utils.miscr Zpip._internal.vcsr Zsourcesr r r r rj Z getLoggerr4 rS r7 ZResponseHeadersr* � Exceptionr+ rB rC rP rV r[ r\ rm ry r� r] r r� rk r� r� r� r� r( r( r( r) �<module> st 4
? �
� ��
=