Merge pull request #439 from seleniumbase/update-reqs-and-ad-blocking-and-methods

Update requirements, ad-blocking settings, and methods

Author: mdmintz

README.md

@@ -1,9 +1,15 @@
-[<img src="https://cdn2.hubspot.net/hubfs/100006/images/super_logo_sb.png" title="SeleniumBase" height="48">](https://github.com/seleniumbase/SeleniumBase/blob/master/README.md)
+[<img src="https://cdn2.hubspot.net/hubfs/100006/images/super_logo_sb7.png" title="SeleniumBase" height="48">](https://github.com/seleniumbase/SeleniumBase/blob/master/README.md)
 
 [<img src="https://img.shields.io/github/release/seleniumbase/SeleniumBase.svg" alt=" " />](https://github.com/seleniumbase/SeleniumBase/releases) [<img src="https://dev.azure.com/seleniumbase/seleniumbase/_apis/build/status/seleniumbase.SeleniumBase?branchName=master" alt=" " />](https://dev.azure.com/seleniumbase/seleniumbase/_build/latest?definitionId=1&branchName=master) [<img src="https://travis-ci.org/seleniumbase/SeleniumBase.svg?branch=master" alt=" " />](https://travis-ci.org/seleniumbase/SeleniumBase) [<img src="https://badges.gitter.im/seleniumbase/SeleniumBase.svg" alt=" " />](https://gitter.im/seleniumbase/SeleniumBase) [<img src="https://img.shields.io/badge/license-MIT-22BBCC.svg" alt=" " />](https://github.com/seleniumbase/SeleniumBase/blob/master/LICENSE) [<img src="https://img.shields.io/github/stars/seleniumbase/seleniumbase.svg" alt=" " />](https://github.com/seleniumbase/SeleniumBase/stargazers)
 
 All-in-one framework for web automation, end-to-end testing, and website tours. SeleniumBase uses pytest for running Python scripts, while using Selenium WebDriver for controlling web browsers.
 
+<img src="https://cdn2.hubspot.net/hubfs/100006/images/my_first_test_gif.gif" title="SeleniumBase"><br />
+(<i>Above: [my_first_test.py](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/my_first_test.py) from [examples/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples) in Demo Mode.</i>)<br />
+```
+pytest my_first_test.py --demo
+```
+
 * Contains reliable, smart-waiting code to prevent flaky tests.
 * Simplifies the process of creating UI tests for any website.
 * Includes [Plugins](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/plugins/pytest_plugin.py) for logging [test results and screenshots](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/example_logs/ReadMe.md).
@@ -14,12 +20,6 @@ All-in-one framework for web automation, end-to-end testing, and website tours.
 * Supports [Azure Pipelines](https://github.com/seleniumbase/SeleniumBase/blob/master/integrations/azure/azure_pipelines/ReadMe.md), [GCP](https://github.com/seleniumbase/SeleniumBase/blob/master/integrations/google_cloud/ReadMe.md), [TravisCI](https://github.com/seleniumbase/SeleniumBase/blob/master/.travis.yml), [Docker](https://github.com/seleniumbase/SeleniumBase/blob/master/integrations/docker/ReadMe.md), and more.
 * To see the full list of SeleniumBase features, [click here](https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/features_list.md).
 
-<img src="https://cdn2.hubspot.net/hubfs/100006/images/my_first_test_gif.gif" title="SeleniumBase"><br />
-(<i>Above: [my_first_test.py](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/my_first_test.py) from [examples/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples) in Demo Mode.</i>)<br />
-```
-pytest my_first_test.py --demo
-```
-
 ## <img src="https://cdn2.hubspot.net/hubfs/100006/images/super_square_logo_3a.png" title="SeleniumBase" height="32"> Get Started:
 
 You'll need **[Python](https://www.python.org/downloads/)** on your System PATH. [<img src="https://img.shields.io/badge/python-2.7,_3.5+-22AADD.svg" alt="Python versions" />](https://www.python.org/downloads/)
@@ -297,7 +297,7 @@ nosetests test_suite.py --report
 ```
 <img src="https://cdn2.hubspot.net/hubfs/100006/images/Test_Report_2.png" title="Example Nosetest Report" height="420">
 
-(NOTE: You can add ``--show_report`` to immediately display Nosetest reports after the test suite completes. Only use ``--show_report`` when running tests locally because it pauses the test run.)
+(NOTE: You can add ``--show-report`` to immediately display Nosetest reports after the test suite completes. Only use ``--show-report`` when running tests locally because it pauses the test run.)
 
 
 ### <img src="https://cdn2.hubspot.net/hubfs/100006/images/super_square_logo_3a.png" title="SeleniumBase" height="32"> **Using a Proxy Server:**
@@ -349,7 +349,7 @@ Here are some things you can do to setup a production environment for your testi
 * If you use [Slack](https://slack.com), you can easily have your Jenkins jobs display results there by using the [Jenkins Slack Plugin](https://github.com/jenkinsci/slack-plugin). Another way to send messages from your tests to Slack is by using [Slack's Incoming Webhooks API](https://api.slack.com/incoming-webhooks).
 
 <a id="amazon_section"></a>
-* If you're using AWS, you can setup an [Amazon S3](https://aws.amazon.com/s3/) account for saving your log files and screenshots for future viewing. SeleniumBase already has [all the code you need to connect to S3](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/plugins/s3_logging_plugin.py). You'll need to modify [settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py) with connection details to your instance and the location in S3 where you want log files to be saved. You'll also need to add "``--with-s3_logging``" on the command-line when you run your tests.
+* If you're using AWS, you can setup an [Amazon S3](https://aws.amazon.com/s3/) account for saving your log files and screenshots for future viewing. SeleniumBase already has [all the code you need to connect to S3](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/plugins/s3_logging_plugin.py). You'll need to modify [settings.py](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/config/settings.py) with connection details to your instance and the location in S3 where you want log files to be saved. You'll also need to add "``--with-s3-logging``" on the command-line when you run your tests.
 
 Here's an example of running tests with additional features enabled:
 ```bash

examples/upload_file_test.py

@@ -8,8 +8,10 @@ class FileUploadButtonTests(BaseCase):
     def test_file_upload_button(self):
         self.open("https://www.w3schools.com/jsref/tryit.asp"
                   "?filename=tryjsref_fileupload_get")
-        self.wait_for_element('[id*="google_ads"]')
-        self.remove_elements('[id*="google_ads"]')
+        if not self.ad_block_on:
+            # If ad-blocking is disabled, block anyway.
+            self.wait_for_element('[id*="google_ads"]')
+            self.remove_elements('[id*="google_ads"]')
         self.switch_to_frame('iframeResult')
         self.add_css_style(
             'input[type="file"]{zoom: 1.5;-moz-transform: scale(1.5);}')

help_docs/features_list.md

@@ -1,4 +1,4 @@
-[<img src="https://cdn2.hubspot.net/hubfs/100006/images/super_logo_sb4.png" title="SeleniumBase" height="48">](https://github.com/seleniumbase/SeleniumBase/blob/master/README.md)
+[<img src="https://cdn2.hubspot.net/hubfs/100006/images/super_logo_sb.png" title="SeleniumBase" height="48">](https://github.com/seleniumbase/SeleniumBase/blob/master/README.md)
 
 <a id="feature_list"></a>
 ## <img src="https://cdn2.hubspot.net/hubfs/100006/images/super_square_logo_3a.png" title="SeleniumBase" height="32"> **Features:**

help_docs/method_summary.md

@@ -173,6 +173,8 @@ self.wait_for_ready_state_complete(timeout=None)
 
 self.wait_for_angularjs(timeout=None)
 
+self.sleep(seconds)
+
 self.activate_jquery()
 
 self.bring_to_front(selector, by=By.CSS_SELECTOR)

requirements.txt

@@ -2,17 +2,17 @@ pip>=19.3.1
 setuptools>=42.0.2
 setuptools-scm>=3.3.3
 wheel>=0.33.6
-six>=1.13.0
-nose>=1.3.7
-ipdb>=0.12.2
+six==1.13.0
+nose==1.3.7
+ipdb==0.12.3
 idna==2.8
 chardet==3.0.4
-urllib3==1.25.6
-requests>=2.22.0
+urllib3==1.25.7
+requests==2.22.0
 selenium==3.141.0
 pluggy>=0.13.1
 attrs>=19.3.0
-pytest>=4.6.6;python_version<"3"
+pytest>=4.6.7;python_version<"3"
 pytest>=5.3.1;python_version>="3"
 pytest-cov>=2.8.1
 pytest-forked>=1.1.3
@@ -24,19 +24,20 @@ pytest-rerunfailures>=8.0
 pytest-timeout>=1.3.3
 pytest-xdist>=1.30.0
 parameterized>=0.7.1
-beautifulsoup4>=4.6.0
-atomicwrites>=1.3.0
-portalocker>=1.5.2
-cryptography>=2.8
-asn1crypto>=1.2.0
-pyopenssl>=19.1.0
-pygments>=2.5.2
-colorama>=0.4.1
-pymysql>=0.9.3
-pyotp>=2.3.0
-boto>=2.49.0
+soupsieve==1.9.5
+beautifulsoup4==4.8.1
+atomicwrites==1.3.0
+portalocker==1.5.2
+cryptography==2.8
+asn1crypto==1.2.0
+pyopenssl==19.1.0
+pygments==2.5.2
+colorama==0.4.1
+pymysql==0.9.3
+pyotp==2.3.0
+boto==2.49.0
 cffi>=1.13.2
-tqdm>=4.40.0
-flake8>=3.7.9
+tqdm>=4.40.1
+flake8==3.7.9
 certifi>=2019.11.28
 pdfminer.six==20191110

seleniumbase/config/ad_block_list.py

@@ -17,8 +17,21 @@
 """
 
 AD_BLOCK_LIST = [
+    'script[src*="/adservice."]',
+    'script[type="data-doubleclick"]',
+    'script[src*="doubleclick.net"]',
+    'script[src*="googletagservices.com/"]',
+    'script[src*="ad.doubleclick.net/"]',
+    'script[src*="adsbygoogle.js"]',
+    'script[src*="adroll.com"]',
+    'script[src*="/pagead/"]',
+    'iframe[onload*="doWithAds"]',
+    'iframe[src*="doubleclick.net"]',
+    'iframe[id*="google_ads_frame"]',
     '[aria-label="Ad"]',
     '[class^="sponsored-content"]',
+    '[class="adsbygoogle"]',
+    '[class^="adroll"]',
     '[data-ad-details*="Advertisement"]',
     '[data-native_ad*="placement"]',
     '[data-provider="dianomi"]',
@@ -28,27 +41,27 @@
     '[data-ad-feedback-beacon]',
     '[href*="doubleclick.net/"]',
     '[href*="amazon-adsystem"]',
+    '[alt="Advertisement"]',
     '[id*="-ad-"]',
     '[id*="_ads_"]',
-    '[alt="Advertisement"]',
     '[id*="AdFrame"]',
     '[id*="carbonads"]',
     '[id^="ad-"]',
     '[id^="my-ads"]',
     '[id^="outbrain_widget"]',
     '[id^="taboola-"]',
+    '[id^="google_ads_frame"]',
     '[id="dianomiRightRail"]',
     '[src*="smartads."]',
     '[src*="ad_nexus"]',
     '[src*="/ads/"]',
+    '[allow*="advertising.com"]',
     '[data-dcm-click-tracker*="/adclick."]',
     '[data-google-query-id^="C"]',
-    '[allow*="advertising.com"]',
     '[data-ylk*="pkgt:sponsored_cluster"]',
+    '[data-google-av-cxn*="pagead2"]',
+    '[data-ad-client]',
     '[data-ad-slot]',
-    'iframe[onload*="doWithAds"]',
-    'iframe[src*="doubleclick.net"]',
-    'iframe[id*="google_ads_frame"]',
     'ins.adsbygoogle',
     'li.strm-ad-clusters',
     'li.js-stream-ad',
@@ -57,19 +70,15 @@
     'div.ad_module',
     'div.ad-subnav-container',
     'div.ad-wrapper',
+    'div.adroll-block',
     'div.data-ad-container',
+    'div.GoogleActiveViewElement',
     'div.l-ad',
     'div.right-ad',
     'div.wx-adWrapper',
     'div.image > a > img[src*="HomepageAd-"]',
     'img[src*="HomepageAd-"]',
     'img.img_ad',
     'link[href*="/adservice."]',
-    'script[src*="/adservice."]',
-    'script[type="data-doubleclick"]',
-    'script[src*="doubleclick.net"]',
-    'script[src*="googletagservices.com/"]',
-    'script[src*="ad.doubleclick.net/"]',
-    'script[src*="/pagead/"]',
     'section.dianomi-ad',
 ]

seleniumbase/fixtures/base_case.py

@@ -1608,6 +1608,9 @@ def wait_for_angularjs(self, timeout=None, **kwargs):
             timeout = self.__get_new_timeout(timeout)
         js_utils.wait_for_angularjs(self.driver, timeout, **kwargs)
 
+    def sleep(self, seconds):
+        time.sleep(seconds)
+
     def activate_jquery(self):
         """ If "jQuery is not defined", use this method to activate it for use.
             This happens because jQuery is not always defined on web sites. """

setup.py

@@ -45,7 +45,7 @@
 
 setup(
     name='seleniumbase',
-    version='1.33.9',
+    version='1.33.10',
     description='Fast, Easy, and Reliable Browser Automation & Testing.',
     long_description=long_description,
     long_description_content_type='text/markdown',
@@ -90,12 +90,12 @@
         'ipdb',
         'idna==2.8',  # Must stay in sync with "requests"
         'chardet==3.0.4',  # Must stay in sync with "requests"
-        'urllib3==1.25.6',  # Must stay in sync with "requests"
-        'requests>=2.22.0',
+        'urllib3==1.25.7',  # Must stay in sync with "requests"
+        'requests==2.22.0',
         'selenium==3.141.0',
         'pluggy>=0.13.1',
         'attrs>=19.3.0',
-        'pytest>=4.6.6;python_version<"3"',  # For Python 2 compatibility
+        'pytest>=4.6.7;python_version<"3"',  # For Python 2 compatibility
         'pytest>=5.3.1;python_version>="3"',
         'pytest-cov>=2.8.1',
         'pytest-forked>=1.1.3',
@@ -107,20 +107,21 @@
         'pytest-timeout>=1.3.3',
         'pytest-xdist>=1.30.0',
         'parameterized>=0.7.1',
-        'beautifulsoup4>=4.6.0',  # Keep at >=4.6.0 while using "bs4"
-        'atomicwrites>=1.3.0',
-        'portalocker>=1.5.2',
-        'cryptography>=2.8',
-        'asn1crypto>=1.2.0',
-        'pyopenssl>=19.1.0',
+        'soupsieve==1.9.5',
+        'beautifulsoup4==4.8.1',
+        'atomicwrites==1.3.0',
+        'portalocker==1.5.2',
+        'cryptography==2.8',
+        'asn1crypto==1.2.0',
+        'pyopenssl==19.1.0',
         'pygments>=2.5.2',
-        'colorama>=0.4.1',
-        'pymysql>=0.9.3',
-        'pyotp>=2.3.0',
-        'boto>=2.49.0',
+        'colorama==0.4.1',
+        'pymysql==0.9.3',
+        'pyotp==2.3.0',
+        'boto==2.49.0',
         'cffi>=1.13.2',
-        'tqdm>=4.40.0',
-        'flake8>=3.7.9',
+        'tqdm>=4.40.1',
+        'flake8==3.7.9',
         'certifi>=2019.11.28',
         'pdfminer.six==20191110',
     ],