RTB
|
5b871de987
|
Merge branch 'feature/sources-configuration' of github.com:Recondor/Fourmi into feature/sources-configuration
|
2014-06-06 23:11:53 +02:00 |
|
Jip J. Dekker
|
c9f902993c
|
Merge branch 'develop' into feature/sources-configuration
|
2014-06-06 22:31:49 +02:00 |
|
Jip J. Dekker
|
321295a395
|
Fixed the synonym requests function for the start requests
|
2014-06-06 22:22:29 +02:00 |
|
RTB
|
30f00b676d
|
updated parse to use newresult function
|
2014-06-06 20:16:25 +02:00 |
|
RTB
|
69664d3ac0
|
added newrresult function to WikipediaParser.py
|
2014-06-06 19:30:31 +02:00 |
|
RTB
|
68139b4839
|
added config to wikipedia along with reliability check
|
2014-06-06 19:27:27 +02:00 |
|
RTB
|
981615c6b3
|
chemspider __init__ now sets reliability to empty string if it does not exist in config
|
2014-06-06 19:14:24 +02:00 |
|
RTB
|
f6f5c5f6fe
|
added config to NIST along with reliability check
|
2014-06-06 19:13:25 +02:00 |
|
RTB
|
a77eafe513
|
updated parse_generic_info to use newresult function
|
2014-06-06 19:08:22 +02:00 |
|
RTB
|
80770de5c0
|
updated parse_aggregate_data to use newresult function
|
2014-06-06 19:06:22 +02:00 |
|
RTB
|
c49d76cb66
|
updated parse_transition_data to use newresult function
|
2014-06-06 19:03:51 +02:00 |
|
RTB
|
ed53889018
|
updated parse_generic_data to use newresult function
|
2014-06-06 19:00:04 +02:00 |
|
RTB
|
a272f9f6d6
|
updated parse_antoine_data to use newresult function
|
2014-06-06 18:58:33 +02:00 |
|
RTB
|
e347b7538d
|
updated parse_individidual_datapoints to use newresult function
|
2014-06-06 18:54:38 +02:00 |
|
RTB
|
de21891bff
|
created newresult function in NIST.py
|
2014-06-06 18:52:18 +02:00 |
|
RTB
|
0021953a9a
|
updated parse_properties to use newresult function
|
2014-06-06 18:24:32 +02:00 |
|
RTB
|
b3b879d2ad
|
updated parse_extendedinfo to use the newresult function
|
2014-06-06 18:24:16 +02:00 |
|
RTB
|
755c981efa
|
created newresult function that uses the config for reliability
|
2014-06-06 18:24:05 +02:00 |
|
RTB
|
217fb3e9cd
|
ChemSpider now uses the token from sources.cfg with checks
|
2014-06-06 16:17:46 +02:00 |
|
RTB
|
df4ba2f784
|
changed __init__ of all sources to have an empty dictionary as default config value
|
2014-06-06 12:48:30 +02:00 |
|
RTB
|
ff3b81b813
|
each source now receives a configuration dictionary
|
2014-06-05 16:30:48 +02:00 |
|
Jip J. Dekker
|
242e0bf628
|
Code inspection
|
2014-06-04 19:43:33 +02:00 |
|
Jip J. Dekker
|
046fbed3cd
|
Code reformat
|
2014-06-04 19:34:23 +02:00 |
|
Jip J. Dekker
|
eb727bd6c4
|
No two requests shall be the same!
|
2014-06-04 19:12:08 +02:00 |
|
Jip J. Dekker
|
0c9862d836
|
Damn you semicolon!
|
2014-06-04 18:54:29 +02:00 |
|
Jip J. Dekker
|
f128c54312
|
Sources don't need to be mangled
|
2014-06-04 18:34:31 +02:00 |
|
Jip J. Dekker
|
75c0be1fea
|
Added tests for the pipline
|
2014-06-04 16:50:14 +02:00 |
|
Jip J. Dekker
|
c48c4ec697
|
None pipeline doesn't need a set
|
2014-06-04 16:09:55 +02:00 |
|
Jip J. Dekker
|
d4a0ffdff3
|
Optimized imports
|
2014-06-04 12:01:05 +02:00 |
|
Jip J. Dekker
|
ecee4a5f45
|
Merge branch 'develop' of github.com:Recondor/Fourmi into develop
|
2014-06-01 20:30:50 +02:00 |
|
Jip J. Dekker
|
aac0a7c79c
|
References to the main Scrapy documentation
|
2014-06-01 20:29:51 +02:00 |
|
Jip J. Dekker
|
f81b1c9500
|
Fixed a typo
|
2014-06-01 20:25:46 +02:00 |
|
Jip J. Dekker
|
f7d0fb4a45
|
Added documentation to the basic Source
|
2014-06-01 20:24:54 +02:00 |
|
Jip J. Dekker
|
c27a875d68
|
Parser/Source consistency
|
2014-06-01 20:18:03 +02:00 |
|
Jip J. Dekker
|
3499946e97
|
Fixed a typo
|
2014-06-01 20:15:15 +02:00 |
|
Jip J. Dekker
|
c4876f029b
|
Added documentation to the FourmiSpider
|
2014-06-01 20:14:47 +02:00 |
|
Jip J. Dekker
|
ace4393a8f
|
Merge branch 'feature/NIST-source' into develop
|
2014-05-23 13:01:06 +02:00 |
|
Jip J. Dekker
|
0e7e4cbe61
|
Merge branch 'develop' of github.com:Recondor/Fourmi into develop
|
2014-05-22 12:17:56 +02:00 |
|
Jip J. Dekker
|
98f91a1aa9
|
Added a pipeline to replace None values with empty strings
|
2014-05-22 12:15:43 +02:00 |
|
Rob tB
|
6ce5ff2335
|
replaced name variable with summary variable
|
2014-05-21 10:40:44 +02:00 |
|
Rob tB
|
6cd8edaf22
|
included summary variable in call to transition_table, antoine table and generic table
|
2014-05-21 10:36:42 +02:00 |
|
Rob tB
|
c0af24644b
|
added summary variable in parse()
|
2014-05-21 10:31:19 +02:00 |
|
Rob tB
|
429ffd7422
|
renamed tables to table in parse()
|
2014-05-21 10:28:54 +02:00 |
|
Rob tB
|
95565042ca
|
removed unused variable symbol_table from parse_transition_table
|
2014-05-21 10:22:03 +02:00 |
|
RTB
|
81719a38fb
|
Added comments for the class and functions
|
2014-05-20 19:32:06 +02:00 |
|
RTB
|
472aae86be
|
synonyms are now scraped
|
2014-05-17 19:32:20 +02:00 |
|
RTB
|
b46c7a309d
|
if synonym name matched in search instead of primary name, emit primary name as synonym
|
2014-05-17 14:21:11 +02:00 |
|
RTB
|
afc1106838
|
NIST now logs an error if chemical name is not found
|
2014-05-17 14:11:03 +02:00 |
|
RTB
|
56ee6b1ad3
|
added ignore list
|
2014-05-17 14:09:10 +02:00 |
|
Rob tB
|
98f58ea4e2
|
added scraping for generic info except for synonyms
|
2014-05-15 14:29:28 +02:00 |
|