Make fetch test work with .onion domains · Pull Request #11988 · EFForg/https-everywhere

ghost · Aug 19, 2017

Requires EFForg/https-everywhere-docker-base#2 to work.

ghost · Aug 19, 2017

cschanaj · Aug 19, 2017

did you try ./fetch-test.sh ./rules/FacebookCoreWWWi.onion.xml locally?

cschanaj · Aug 19, 2017

 import traceback
 import subprocess
 import re
+from urlparse import urlparse


cschanaj · Aug 19, 2017

 			buf = cStringIO.StringIO()
 			headerBuf = cStringIO.StringIO()

+			urlParts = urlparse(url)


maybe urlParts = urlparse.urlparse(url)?

ghost · Aug 19, 2017

@cschanaj No, for that I need @Hainish to update Docker base.

ghost · Aug 19, 2017

@cschanaj Done.

cschanaj · Aug 19, 2017

I get the following result on my local machine with tor installed. Not really sure why though.

$ ./fetch-test.sh ./rules/FacebookCoreWWWi.onion.xml 
INFO Finished comparing http://developers.facebookcorewwwi.onion/ -> https://developers.facebookcorewwwi.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://m.facebookcorewwwi.onion/ -> https://m.facebookcorewwwi.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://www.facebookcorewwwi.onion/ -> https://www.facebookcorewwwi.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://facebookcorewwwi.onion/ -> https://facebookcorewwwi.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://scontent.xx.fbcdn23dssr3jqnq.onion/ -> https://scontent.xx.fbcdn23dssr3jqnq.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
ERROR ./rules/FacebookCoreWWWi.onion.xml: Fetch error: http://developers.facebookcorewwwi.onion/ => https://developers.facebookcorewwwi.onion/: CURLOPT_PROXY
ERROR ./rules/FacebookCoreWWWi.onion.xml: Fetch error: http://m.facebookcorewwwi.onion/ => https://m.facebookcorewwwi.onion/: CURLOPT_PROXY
ERROR ./rules/FacebookCoreWWWi.onion.xml: Fetch error: http://www.facebookcorewwwi.onion/ => https://www.facebookcorewwwi.onion/: CURLOPT_PROXY
ERROR ./rules/FacebookCoreWWWi.onion.xml: Fetch error: http://facebookcorewwwi.onion/ => https://facebookcorewwwi.onion/: CURLOPT_PROXY
ERROR ./rules/FacebookCoreWWWi.onion.xml: Fetch error: http://scontent.xx.fbcdn23dssr3jqnq.onion/ => https://scontent.xx.fbcdn23dssr3jqnq.onion/: CURLOPT_PROXY
INFO Finished in 0.01 seconds. Loaded rulesets: 1, URL pairs: 5.

Also, I expect the timeout value needed to be increased.

ghost · Aug 19, 2017

@cschanaj Was Tor running?

ghost · Aug 19, 2017

@cschanaj Try curl -v --socks5-hostname localhost:9050 https://facebookcorewwwi.onion

cschanaj · Aug 19, 2017

curl -v --socks5-hostname 127.0.0.1:9050 https://facebookcorewwwi.onion works, I guess it is something related to pycurl

cschanaj · Aug 19, 2017

 			c.setopt(c.VERBOSE, options.curlVerbose)
 			c.setopt(c.SSL_CIPHER_LIST, options.cipherList)
+			if urlParts.hostname[-6:] == '.onion':
+				c.setopt(c.CURLOPT_PROXY, '127.0.0.1:9050')


please replace line 331 and 332 with the following

c.setopt(c.PROXY, 'socks5h://127.0.0.1:9050')

or

c.setopt(c.PROXY, '127.0.0.1') c.setopt(c.PROXYPORT, 9050) c.setopt(c.PROXYTYPE, c.PROXYTYPE_SOCKS5_HOSTNAME)

this typo is so difficult to trace...

@cschanaj Done. Working now?

cschanaj · Aug 19, 2017

@koops76 AFAIK, it works on my local machine and finished the fetch-test with exit 0. Please run your code at least locally next time, since fetch-test is not triggered by the core code changes.

Linking #11187 (no conflict at the moment)

ghost · Aug 19, 2017

Try testing a deliberately broken ruleset like replace the hostname with doesntexistdoesn.onion.

cschanaj · Aug 19, 2017

Not really sure if this is good enough. I'd like to leave this for @Hainish to decide. thanks.

$ ./fetch-test.sh ./rules/FacebookCoreWWWi.onion.xml 
INFO Finished comparing http://developers.facebookcorewwwi.onion/ -> https://developers.facebookcorewwwi.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://m.facebookcorewwwi.onion/ -> https://m.facebookcorewwwi.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://www.facebookcorewwwi.onion/ -> https://www.facebookcorewwwi.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://doesntexistdoesn.onion/ -> https://doesntexistdoesn.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://facebookcorewwwi.onion/ -> https://facebookcorewwwi.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
INFO Finished comparing http://scontent.xx.fbcdn23dssr3jqnq.onion/ -> https://scontent.xx.fbcdn23dssr3jqnq.onion/. Rulefile: ./rules/FacebookCoreWWWi.onion.xml.
ERROR ./rules/FacebookCoreWWWi.onion.xml: Fetch error: http://doesntexistdoesn.onion/ => https://doesntexistdoesn.onion/: (7, "Can't complete SOCKS5 connection to 0.0.0.0:0. (4)")
INFO Finished in 63.31 seconds. Loaded rulesets: 1, URL pairs: 6.

P.S. I do not want to get too involved in this since I have other priorities in this project. Thanks for this PR anyway!!

J0WI · Aug 20, 2017

@koops76 the Docker base can be updated here: https://github.com/EFForg/https-everywhere-docker-base/blob/master/Dockerfile

Edit: Never mind, just found your PR there.

Hainish · Aug 23, 2017

@koops76 you can test locally using docker. Just run

sudo docker build -t electronicfrontierfoundation/https-everywhere-docker-base .

Once that image is built on your system, you can then run in this project

sudo docker build -t httpse .
sudo docker run --rm -ti -v $(pwd):/opt httpse bash -c "./fetch-test.sh ./rules/FacebookCoreWWWi.onion.xml"

ghost · Aug 23, 2017

@Hainish I'm not running Linux, so I can't test it.

Hainish · Aug 25, 2017

Thanks for this!

ghost · Aug 25, 2017

@Hainish See #11806.

koops76 added 2 commits August 19, 2017 16:14

Update fetch.sh

f33b317

Update http_client.py

cdde174

ghost changed the title ~~Update fetch.sh~~ Make fetch test work with .onion domains Aug 19, 2017

koops76 added 3 commits August 19, 2017 17:29

Update fetch.sh

1ef7b7d

Update fetch.sh

a0666a0

Update travis.sh

3073c8a

This was referenced Aug 19, 2017

Remove platform from *.onion rulesets #11806

Closed

[Idea] Tor Onion services #3798

Closed

cschanaj reviewed Aug 19, 2017

View reviewed changes

Update http_client.py

0271b44

cschanaj reviewed Aug 19, 2017

View reviewed changes

Update http_client.py

488c70d

Update http_client.py

f459b69

J0WI added the ruleset-testing label Aug 20, 2017

Hainish merged commit e2f93d0 into EFForg:master Aug 25, 2017

ghost deleted the patch-5 branch August 25, 2017 23:32

Search code, repositories, users, issues, pull requests...

Uh oh!

Conversation

ghost commented Aug 19, 2017 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Aug 19, 2017

Uh oh!

cschanaj commented Aug 19, 2017

Uh oh!

cschanaj Aug 19, 2017

Choose a reason for hiding this comment

Uh oh!

cschanaj Aug 19, 2017

Choose a reason for hiding this comment

Uh oh!

ghost commented Aug 19, 2017

Uh oh!

ghost commented Aug 19, 2017

Uh oh!

cschanaj commented Aug 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Aug 19, 2017

Uh oh!

ghost commented Aug 19, 2017

Uh oh!

cschanaj commented Aug 19, 2017

Uh oh!

cschanaj Aug 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost Aug 19, 2017

Choose a reason for hiding this comment

Uh oh!

cschanaj commented Aug 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Aug 19, 2017 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cschanaj commented Aug 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

J0WI commented Aug 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hainish commented Aug 23, 2017

Uh oh!

ghost commented Aug 23, 2017

Uh oh!

Hainish commented Aug 25, 2017

Uh oh!

ghost commented Aug 25, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ghost commented Aug 19, 2017 •

edited by ghost

Loading

cschanaj commented Aug 19, 2017 •

edited

Loading

cschanaj Aug 19, 2017 •

edited

Loading

cschanaj commented Aug 19, 2017 •

edited

Loading

ghost commented Aug 19, 2017 •

edited by ghost

Loading

cschanaj commented Aug 19, 2017 •

edited

Loading

J0WI commented Aug 20, 2017 •

edited

Loading