a Oc\O@sLdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl ZddlmZddlmZddlmZmZmZmZmZmZmZmZmZmZmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%dd l&m'Z'dd l(m)Z)dd l*m+Z+dd l,m-Z-dd l.m/Z/ddl0m1Z1ddl2m3Z3m4Z4ddl5m6Z6ddl7m8Z8m9Z9m:Z:er\ddlm;Z;neZ?ej@jAjBZCeeDeDfZEeDeeDdddZFGdddeGZHe"ddddZIGdddeGZJeDe-dddd ZKeDe-e"dd!d"ZLeEeeDd#d$d%ZMeDeDd&d'd(ZNeDeDd&d)d*ZOe Pd+e jQZReDeSeDd,d-d.ZTeDeDdd/d0ZUeeDeeDfeDeDee)d1d2d3ZVGd4d5d5ZWGd6d7d7e;ZXeXeXd8d9d:ZYeYd;ee)d<d=d>ZZGd?d;d;Z[Gd@dAdAeZ\dQe)eeDeGfeedBddCdDdEZ]dRe"eSe[dGdHdIZ^dSe)ee-ed;dJdKdLZ_GdMdNdNeZ`GdOdPdPZadS)TzO The main purpose of this module is to expose LinkCollector.collect_sources(). N) HTMLParser)Values) TYPE_CHECKINGCallableDictIterableListMutableMapping NamedTupleOptionalSequenceTupleUnion)requests)Response) RetryErrorSSLError)NetworkConnectionError)Link) SearchScope) PipSession)raise_for_status)is_archive_file)pairwiseredact_auth_from_url)vcs)CandidatesFromPage LinkSource build_source)ProtocolurlreturncCs6tjD]*}||r|t|dvr|SqdS)zgLook for VCS schemes in the URL. Returns the matched VCS scheme, or None if there's no match. z+:N)rZschemeslower startswithlen)r"schemer(U/var/www/html/megastudio/lib/python3.9/site-packages/pip/_internal/index/collector.py_match_vcs_scheme:s  r*cs&eZdZeeddfdd ZZS)_NotAPIContentN) content_type request_descr#cst||||_||_dSN)super__init__r,r-)selfr,r- __class__r(r)r0Fsz_NotAPIContent.__init__)__name__ __module__ __qualname__strr0 __classcell__r(r(r2r)r+Esr+)responser#cCs6|jdd}|}|dr$dSt||jjdS)z Check the Content-Type header to ensure the response contains a Simple API Response. Raises `_NotAPIContent` if the content type is not a valid content-type. Content-TypeUnknown)z text/htmlz#application/vnd.pypi.simple.v1+html#application/vnd.pypi.simple.v1+jsonN)headersgetr$r%r+requestmethod)r9r,content_type_lr(r(r)_ensure_api_headerLsrBc@s eZdZdS)_NotHTTPN)r4r5r6r(r(r(r)rCbsrC)r"sessionr#cCsFtj|\}}}}}|dvr$t|j|dd}t|t|dS)z Send a HEAD request to the URL, and ensure the response contains a simple API Response. Raises `_NotHTTP` if the URL is not available for a HEAD request, or `_NotAPIContent` if the content type is not a valid content type. >httpshttpT)allow_redirectsN)urllibparseurlsplitrCheadrrB)r"rDr'netlocpathqueryfragmentrespr(r(r)_ensure_api_responsefs rQcCsxtt|jrt||dtdt||j|dgdddd}t |t |tdt||j d d |S) aYAccess an Simple API response with GET, and return the response. This consists of three parts: 1. If the URL looks suspiciously like an archive, send a HEAD first to check the Content-Type is HTML or Simple API, to avoid downloading a large file. Raise `_NotHTTP` if the content type cannot be determined, or `_NotAPIContent` if it is not HTML or a Simple API. 2. Actually perform the request. Raise HTTP exceptions on network failures. 3. Check the Content-Type header to make sure we got a Simple API response, and raise `_NotAPIContent` otherwise. rDzGetting page %sz, )r<z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)Acceptz Cache-Control)r=zFetched page %s as %sr:r;) rrfilenamerQloggerdebugrr>joinrrBr=)r"rDrPr(r(r)_get_simple_responsexs&   rX)r=r#cCs<|r8d|vr8tj}|d|d<|d}|r8t|SdS)z=Determine if we have any encoding information in our headers.r:z content-typecharsetN)emailmessageMessage get_paramr7)r=mrYr(r(r)_get_encoding_from_headerss    r_)partr#cCstjtj|S)zP Clean a "part" of a URL path (i.e. after splitting on "@" characters). )rHrIquoteunquoter`r(r(r)_clean_url_path_partsrdcCstjtj|S)z Clean the first part of a URL path that corresponds to a local filesystem path (i.e. the first part after splitting on "@" characters). )rHr? pathname2url url2pathnamercr(r(r)_clean_file_url_paths rgz(@|%2F))rM is_local_pathr#cCs^|r t}nt}t|}g}tt|dgD]$\}}|||||q.d |S)z* Clean the path portion of a URL. ) rgrd_reserved_chars_resplitr itertoolschainappendupperrW)rMrhZ clean_funcpartsZ cleaned_partsZto_cleanreservedr(r(r)_clean_url_paths rrcCs6tj|}|j }t|j|d}tj|j|dS)z Make sure a link is fully quoted. For example, if ' ' occurs in the URL, it will be replaced with "%20", and without double-quoting other characters. )rh)rM)rHrIurlparserLrrrM urlunparse_replace)r"resultrhrMr(r(r) _clean_links rw)element_attribspage_urlbase_urlr#cCsL|d}|sdSttj||}|d}|d}t||||d}|S)zW Convert an anchor element's attributes in a simple repository page to a Link. hrefNzdata-requires-pythonz data-yanked) comes_fromrequires_python yanked_reason)r>rwrHrIurljoinr)rxryrzr{r"Z pyrequirer~linkr(r(r)_create_link_from_elements   rc@s:eZdZdddddZeedddZed d d ZdS) CacheablePageContent IndexContentNpager#cCs|js J||_dSr.)cache_link_parsingrr1rr(r(r)r0s zCacheablePageContent.__init__)otherr#cCst|t|o|jj|jjkSr.) isinstancetyperr")r1rr(r(r)__eq__szCacheablePageContent.__eq__r#cCs t|jjSr.)hashrr"r1r(r(r)__hash__"szCacheablePageContent.__hash__) r4r5r6r0objectboolrintrr(r(r(r)rsrc@s eZdZdeedddZdS) ParseLinksrrcCsdSr.r(rr(r(r)__call__'szParseLinks.__call__N)r4r5r6rrrr(r(r(r)r&sr)fnr#csLtjddtttdfdd tdttdfdd }|S) z Given a function that parses an Iterable[Link] from an IndexContent, cache the function's result (keyed by CacheablePageContent), unless the IndexContent `page` has `page.cache_link_parsing == False`. N)maxsize)cacheable_pager#cst|jSr.)listr)r)rr(r)wrapper2sz*with_cached_index_content..wrapperrrcs|jrt|St|Sr.)rrr)rrrr(r)wrapper_wrapper6s z2with_cached_index_content..wrapper_wrapper) functools lru_cacherrrwraps)rrr(rr)with_cached_index_content+s  rrrc cs|j}|drt|j}|dgD]r}|d}|durDq,|d}|rbt|tsbd}n|sjd}t t t j |j||j|d||did Vq,dSt|j}|jpd }||j||j}|jp|} |jD]"} t| || d } | durq| VqdS) z\ Parse a Simple API's Index Content, and yield its anchor elements as Link objects. r<filesr"NZyankedrizrequires-pythonhashes)r|r}r~rzutf-8)ryrz)r,r$r%jsonloadscontentr>rr7rrwrHrIrr"HTMLLinkParserencodingfeeddecoderzanchorsr) rrAdatafileZfile_urlr~parserrr"rzanchorrr(r(r) parse_links?sD          rc@s<eZdZdZd eeeeeeddddZeddd Z dS) rz5Represents one response (or page), along with its URLTN)rr,rr"rr#cCs"||_||_||_||_||_dS)am :param encoding: the encoding to decode the given content. :param url: the URL from which the HTML was downloaded. :param cache_link_parsing: whether links parsed from this page's url should be cached. PyPI index urls should have this set to False, for example. N)rr,rr"r)r1rr,rr"rr(r(r)r0rs zIndexContent.__init__rcCs t|jSr.)rr"rr(r(r)__str__szIndexContent.__str__)T) r4r5r6__doc__bytesr7r rr0rr(r(r(r)ros csneZdZdZeddfdd ZeeeeeefddddZ eeeeefeed d d Z Z S) rzf HTMLParser that keeps the first base HREF and a list of all anchor elements' attributes. Nr!cs$tjdd||_d|_g|_dS)NT)Zconvert_charrefs)r/r0r"rzr)r1r"r2r(r)r0szHTMLLinkParser.__init__)tagattrsr#cCsH|dkr,|jdur,||}|durD||_n|dkrD|jt|dS)Nbasea)rzget_hrefrrndict)r1rrr{r(r(r)handle_starttags  zHTMLLinkParser.handle_starttag)rr#cCs"|D]\}}|dkr|SqdS)Nr{r()r1rnamevaluer(r(r)rs  zHTMLLinkParser.get_href) r4r5r6rr7r0rr r rrr8r(r(r2r)rs"r).N)rreasonmethr#cCs|durtj}|d||dS)Nz%Could not fetch URL %s: %s - skipping)rUrV)rrrr(r(r)_handle_get_simple_failsrT)r9rr#cCs&t|j}t|j|jd||j|dS)Nr:)rr"r)r_r=rrr")r9rrr(r(r)_make_index_contents r)rrDr#c Cs|durtd|jddd}t|}|r@td||dStj|\}}}}}}|dkrt j tj |r|ds|d7}tj|d}td |zt||d }WnNtytd |Yn>ty }z"td ||j|jWYd}~n d}~0ty:}zt||WYd}~nd}~0tyh}zt||WYd}~nd}~0ty}z,d } | t|7} t|| tjdWYd}~nld}~0tjy}zt|d|WYd}~n6d}~0tjyt|dYn0t||j dSdS)Nz?_get_html_page() missing 1 required keyword argument: 'session'#rrzICannot look at %s URL %s because it does not support lookup as web pages.r/z index.htmlz# file: URL is directory, getting %srRz`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.zSkipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )rzconnection error: z timed out)r)! TypeErrorr"rkr*rUwarningrHrIrsosrMisdirr?rfendswithrrVrXrCr+r-r,rrrrr7inforConnectionErrorTimeoutrr) rrDr"Z vcs_schemer'_rMrPexcrr(r(r)_get_index_contents^     $$rc@s.eZdZUeeeed<eeeed<dS)CollectedSources find_links index_urlsN)r4r5r6r r r__annotations__r(r(r(r)rs rc@sxeZdZdZeeddddZedeee dddd Z e e e d d d Zeeed ddZe eedddZdS) LinkCollectorz Responsible for collecting Link objects from all configured locations, making network requests as needed. The class's main method is its collect_sources() method. N)rD search_scoper#cCs||_||_dSr.)rrD)r1rDrr(r(r)r0szLinkCollector.__init__F)rDoptionssuppress_no_indexr#cCs`|jg|j}|jr8|s8tdddd|Dg}|jp@g}tj||d}t ||d}|S)z :param session: The Session to use to make requests. :param suppress_no_index: Whether to ignore the --no-index option when constructing the SearchScope object. zIgnoring indexes: %s,css|]}t|VqdSr.)r).0r"r(r(r) (z'LinkCollector.create..rr)rDr) index_urlextra_index_urlsno_indexrUrVrWrrcreater)clsrDrrrrrlink_collectorr(r(r)rs"   zLinkCollector.creatercCs|jjSr.)rrrr(r(r)r9szLinkCollector.find_links)locationr#cCst||jdS)z> Fetch an HTML page containing package links. rR)rrD)r1rr(r(r)fetch_response=szLinkCollector.fetch_response) project_namecandidates_from_pager#cstfddj|D}tfddjD}ttj rddt ||D}t |d|dg|}t d|tt|t|d S) Nc3s$|]}t|jjdddVqdS)FrZpage_validatorZ expand_dirrNrrDZis_secure_originrlocrr1r(r)rIsz0LinkCollector.collect_sources..c3s$|]}t|jjdddVqdS)TrNrrrr(r)rSscSs*g|]"}|dur|jdurd|jqS)Nz* )r)rsr(r(r) _sz1LinkCollector.collect_sources..z' location(s) to search for versions of : r) collections OrderedDictrZget_index_urls_locationsvaluesrrU isEnabledForloggingDEBUGrlrmr&rVrWrr)r1rrZindex_url_sourcesZfind_links_sourceslinesr(rr)collect_sourcesCs*    zLinkCollector.collect_sources)F)r4r5r6rrrr0 classmethodrrrpropertyrr7rrr rrrrrr(r(r(r)rs(   r)N)T)N)brr email.messagerZrrlrrrre urllib.parserHurllib.requestZxml.etree.ElementTreexmlZ html.parserroptparsertypingrrrrrr r r r r r pip._vendorrZpip._vendor.requestsrZpip._vendor.requests.exceptionsrrpip._internal.exceptionsrpip._internal.models.linkrZ!pip._internal.models.search_scoperpip._internal.network.sessionrZpip._internal.network.utilsrpip._internal.utils.filetypesrpip._internal.utils.miscrrpip._internal.vcsrsourcesrrrr r getLoggerr4rUZetreeZ ElementTreeZElementZ HTMLElementr7ZResponseHeadersr* Exceptionr+rBrCrQrXr_rdrgcompile IGNORECASErjrrrrwrrrrrrrrrrrrr(r(r(r)s  4             ?   /     D