File: //usr/local/lib/python3.10/test/__pycache__/test_robotparser.cpython-310.opt-1.pyc
o
�i�+ � @ s( d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlm Z d dlm
Z
d dl m Z m
Z
G dd� d�ZG dd � d eej�ZG d
d � d eej�ZG d d
� d
eej�ZG dd� deej�ZG dd� de�ZG dd� deej�ZG dd� deej�ZG dd� de�ZG dd� deej�ZG dd� deej�ZG d d � d eej�ZG d d � d eej�ZG d d!� d!e�Z G d"d#� d#eej�Z G d$d%� d%eej�Z G d&d'� d'eej�Z G d(d)� d)eej�Z G d*d+� d+eej�Z!G d,d-� d-eej�Z"G d.d/� d/e �Z#G d0d1� d1ej�Z$G d2d3� d3ej�Z%e&d4k�re�'� dS dS )5� N)�support)�
socket_helper)�threading_helper)�BaseHTTPRequestHandler�
HTTPServerc @ sH e Zd ZdZdZg Zg ZdZdd� Zdd� Z dd � Z
d
d � Z d d
� Z dS )�
BaseRobotTest� Ztest_robotparserNc C s, t �| j��� }tj�� | _| j�|� d S �N) �io�StringIO�
robots_txt� readlines�urllib� robotparser�RobotFileParser�parser�parse)�self�lines� r �2/usr/local/lib/python3.10/test/test_robotparser.py�setUp s zBaseRobotTest.setUpc C s$ t |t�r
|\}}||fS | j|fS r )�
isinstance�tuple�agent�r �urlr r r r �get_agent_and_url s
z BaseRobotTest.get_agent_and_urlc C �` | j D ]*}| �|�\}}| j||d�� | �| j�||�� W d � n1 s(w Y qd S �N)r r )�goodr �subTest�
assertTruer � can_fetchr r r r �test_good_urls �
���z BaseRobotTest.test_good_urlsc C r r )�badr r! � assertFalser r# r r r r �
test_bad_urls$ r% zBaseRobotTest.test_bad_urlsc C s | � | j�� | j� d S r )� assertEqualr � site_maps�r r r r �test_site_maps* � z BaseRobotTest.test_site_maps)
�__name__�
__module__� __qualname__r r r r& r* r r r$ r( r, r r r r r s r c @ s e Zd ZdZddgZg d�ZdS )�UserAgentWildcardTestz�User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
�/�
/test.html)�/cyberworld/map/index.htmlz/tmp/xxx� /foo.htmlN�r. r/ r0 r r r& r r r r r1 . s r1 c @ s e Zd ZdZg d�ZdgZdS )� CrawlDelayAndCustomAgentTestz�# robots.txt for http://www.example.com/
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
)r2 r3 )Z cybermapperr4 r4 Nr6 r r r r r7 9 s
r7 c @ s& e Zd ZdZddgZdgZddgZdS )� SitemapTesta # robots.txt for http://www.example.com/
User-agent: *
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
r2 r3 r4 z7http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xmlz2http://www.google.com/hostednews/sitemap_index.xmlN)r. r/ r0 r r r& r* r r r r r8 J s
�r8 c @ s e Zd ZdZg Zg d�ZdS )�RejectAllRobotsTestz(# go away
User-agent: *
Disallow: /
)r4 r2 �/tmp/Nr6 r r r r r9 [ s r9 c @ s e Zd ZdZdZdd� ZdS )�BaseRequestRateTestNc C s� | j }| j| j D ]U}| �|�\}}| j||d��= | �|�|�| j� |�|�}| �|| j� | jd urO| �|t j
j � | �|j | jj � | �|j
| jj
� W d � n1 sYw Y q d S r )r r r& r r! r) � crawl_delay� request_rateZassertIsInstancer r � RequestRateZrequestsZseconds)r r r r Zparsed_request_rater r r �test_request_ratei s0
���� ���z%BaseRequestRateTest.test_request_rate)r. r/ r0 r= r< r? r r r r r; e s r; c @ s e Zd ZdZdgZdS )�
EmptyFileTestr z/fooN)r. r/ r0 r r r r r r r@ � s
r@ c @ s4 e Zd ZdZdZej�dd�ZdZ dgZ
g d�Z dS ) � CrawlDelayAndRequestRateTestz�User-agent: figtree
Crawl-delay: 3
Request-rate: 9/30
Disallow: /tmp
Disallow: /a%3cd.html
Disallow: /a%2fb.html
Disallow: /%7ejoe/index.html
�figtree� � � )rB r5 )�/tmpz /tmp.html� /tmp/a.html� /a%3cd.html� /a%3Cd.htmlz /a%2fb.htmlz/~joe/index.htmlN) r. r/ r0 r r r r r> r= r< r r& r r r r rA � s rA c @ � e Zd ZdZdS )�DifferentAgentTestz FigTree Robot libwww-perl/5.04N�r. r/ r0 r r r r r rK � � rK c @ s" e Zd ZdZdgZg d�ZdZdS )�InvalidRequestRateTestz�User-agent: *
Disallow: /tmp/
Disallow: /a%3Cd.html
Disallow: /a/b.html
Disallow: /%7ejoe/index.html
Crawl-delay: 3
Request-rate: 9/banana
rF )r: rG rH rI z /a/b.htmlz/%7Ejoe/index.htmlrE N)r. r/ r0 r r r& r< r r r r rN � s
rN c @ s e Zd ZdZdgZg ZdS )�InvalidCrawlDelayTestz2User-Agent: *
Disallow: /.
Crawl-delay: pears
r5 Nr6 r r r r rO � s rO c @ � e Zd ZdZdZdgZdgZdS )� AnotherInvalidRequestRateTestzeUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Request-rate: whale/banana
� Googlebot�/folder1/myfile.html�/folder1/anotherfile.htmlN�r. r/ r0 r r r r& r r r r rQ � s
rQ c @ s e Zd ZdZdZdgZdS )�UserAgentOrderingTestzMUser-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
rR z/something.jpgN)r. r/ r0 r r r& r r r r rV � s
rV c @ rJ )�UserAgentGoogleMobileTestzGooglebot-MobileNrL r r r r rW � rM rW c @ rP )�GoogleURLOrderingTestzJUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Z googlebotrS rT NrU r r r r rX � s
rX c @ � e Zd ZdZdgZdgZdS )�DisallowQueryStringTestz2User-agent: *
Disallow: /some/path?name=value
�
/some/pathz/some/path?name=valueNr6 r r r r rZ � s
rZ c @ rY )� UseFirstUserAgentWildcardTestzNUser-agent: *
Disallow: /some/path
User-agent: *
Disallow: /another/path
z
/another/pathr[ Nr6 r r r r r\ � s
r\ c @ rY )�EmptyQueryStringTestz>User-agent: *
Allow: /some/path?
Disallow: /another/path?
z /some/path?z/another/path?Nr6 r r r r r] � s
r] c @ s0 e Zd ZdZej�dd�ZdZddgZ dgZ
dS ) �DefaultEntryTestzOUser-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/
rE � � r2 r3 r4 N) r. r/ r0 r r r r> r= r< r r& r r r r r^ s
r^ c @ s e Zd ZdZdZdd� ZdS )�StringFormattingTestz�User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow: /some/path
zxUser-agent: cybermapper
Disallow: /some/path
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/c C s | � t| j�| j� d S r )r) �strr �expected_outputr+ r r r �test_string_formatting* r- z+StringFormattingTest.test_string_formattingN)r. r/ r0 r rc rd r r r r ra s
ra c @ s e Zd Zdd� Zdd� ZdS )� RobotHandlerc C s | � dd� d S )Ni� zForbidden access)Z
send_errorr+ r r r �do_GET0 s zRobotHandler.do_GETc G s d S r r )r �format�argsr r r � log_message3 s zRobotHandler.log_messageN)r. r/ r0 rf ri r r r r re . s re c @ s* e Zd Zdd� Zdd� Zejdd� �ZdS )� PasswordProtectedSiteTestCasec C sP | � tjj� ttjdft�| _t j
d| jj ddid�| _ d| j _
| j �� d S )Nr zHTTPServer servingZ
poll_intervalg{�G�z�?)�name�target�kwargsT)Z
addCleanupr Zrequest�
urlcleanupr r �HOSTre �server� threading�ThreadZ
serve_forever�t�daemon�startr+ r r r r 9 s �z#PasswordProtectedSiteTestCase.setUpc C s" | j �� | j�� | j �� d S r )rp �shutdownrs �joinZ server_closer+ r r r �tearDownI s
z&PasswordProtectedSiteTestCase.tearDownc C s\ | j j}dtj d t|d � }|d }tj�� }|�|� |� � | �
|� d|�� d S )Nzhttp://�:r` z /robots.txt�*) rp Zserver_addressr ro rb r r r Zset_url�readr' r# )r �addrr Z
robots_urlr r r r �testPasswordProtectedSiteN s
z7PasswordProtectedSiteTestCase.testPasswordProtectedSiteN)r. r/ r0 r rx r Z reap_threadsr} r r r r rj 7 s
rj c @ sF e Zd ZdZd�e�Zedd� �Zdd� Zdd� Z d d
� Z
d d � Z d
S )�NetworkTestCasezhttp://www.pythontest.net/z{}elsewhere/robots.txtc C sV t �d� t�| j�� tj�| j�| _ | j �
� W d � d S 1 s$w Y d S )NZnetwork) r Zrequiresr Ztransient_internet�base_urlr r r r r r{ )�clsr r r �
setUpClass^ s
"�zNetworkTestCase.setUpClassc C s& d� | j|tj�|�d sd�S d�S )Nz{}{}{}r` r2 r )rg r �os�path�splitext)r r� r r r r e s
��zNetworkTestCase.urlc C sV | � | jj� | � | jj� | �| j�� d� | � | j�d�� | � | j�d�� d S )Nr rz )r' r � disallow_all� allow_allZ
assertGreater�mtimer< r= r+ r r r �
test_basicj s
zNetworkTestCase.test_basicc C s� | � | j�d| �d��� | �| j�d| j�� | �| j�d| �d��� | �| j�d| �d��� | �| j�d| �d��� | � | j�d| j�� d S )Nrz Z elsewhereZNutchZbrianZwebstats)r" r r# r r' r r+ r r r �test_can_fetchq s z NetworkTestCase.test_can_fetchc C sf t j�| �d��}|�� | �|j� | �|j� | � |�
� d� | � |� d�� | � |�
d�� d S )Nz i-robot.txtr rz )r r r r r{ r" r� r' r� r) r� Z assertIsNoner< r= )r r r r r �
test_read_404y s z NetworkTestCase.test_read_404N) r. r/ r0 r rg r � classmethodr� r r� r� r� r r r r r~ Y s
r~ �__main__)(r
r� rq ZunittestZurllib.robotparserr �testr Z test.supportr r Z http.serverr r r ZTestCaser1 r7 r8 r9 r; r@ rA rK rN rO rQ rV rW rX rZ r\ r] r^ ra re rj r~ r. �mainr r r r �<module> sF "
"
) �