{"id":47038451,"title":"Tadpole the Language for Scraping 0.2.0 – Complex Control Flow, Stealth and More","points":5,"user":"zachperkitny","time":1771266910,"time_ago":"7 hours ago","type":"link","content":"

Hello,

I posted a few weeks ago about my custom scraping language. It definitely got some traction, which was very exciting to see.

Github Repo: https://github.com/tadpolehq/tadpole\nDocs: https://tadpolehq.com/

The past 2 weeks, I've been focusing my efforts in introducing specific stealth actions, more complicated control flow actions and a lot of various evaluators for cleaning data.

Here is an example for scraping from `books.toscrape.com`

  main {\n    new_page {\n      goto "https://books.toscrape.com/"\n      loop {\n        do {\n          $$ article.product_pod {\n            extract "books[]" {\n              title { $ "h3 a"; attr title }\n              rating {\n                $ ".star-rating";\n                attr "class";\n                extract "star-rating (One|Two|Three|Four|Five)" caseInsensitive=#true;\n                func "(v) => ({'one': 1, 'two': 2, 'three': 3, 'four': 4, 'five': 5}[v.toLowerCase()] || null)"\n              }\n              price { $ "p.price_color"; text; as_float }\n              in_stock { $ "p.availability"; text; matches "In stock" caseInsensitive=#true }\n            }\n          }\n        }\n        while { $ "li.next" }\n        next {\n          $ "li.next a" { click }\n          wait_until\n        }\n      }\n    }\n  }\n

\nI've introduced actions like `apply_identity` to override User Agent Headers and User Agent Metadata. Here is an example module to selectively create different identities:

  module stealth {\n    // Apple M2 Pro\n    action apply_apple_m2 {\n      apply_identity mac\n      set_webgl_vendor "Apple Inc." "Apple M2"\n      set_device_memory 16\n      set_hardware_concurrency 8\n      set_viewport 1440 900 deviceScaleFactor=2\n    }\n\n    // Windows Desktop\n    action apply_windows_16_8 {\n      apply_identity windows\n      set_webgl_vendor "Google Inc. (Intel)" "ANGLE (Intel, Intel(R) UHD Graphics 620 Direct3D11 vs_5_0 ps_5_0)"\n      set_device_memory 16\n      set_hardware_concurrency 8\n      set_viewport 1920 1080\n    }\n\n    // Windows Budget Laptop\n    action apply_windows_8_4 {\n      apply_identity windows\n      set_webgl_vendor "Google Inc. (Intel)" "ANGLE (Intel, Intel(R) UHD Graphics 620 Direct3D11 vs_5_0 ps_5_0)"\n      set_device_memory 8\n      set_hardware_concurrency 4\n      set_viewport 1366 768\n    }\n  }\n\n

\nThe full release changelog is available here:\nhttps://github.com/tadpolehq/tadpole/releases/

My goals for the next 0.3.0 release is to heavily focus on Plugins, Distributed Execution through Message Queues, Redis Support for Crawling, Static Parsing as opposed to exclusively over CDP/Chrome.

I will keep trying to keep my release cadence at every 2 weeks!","url":"item?id=47038451","comments":[],"comments_count":0}