AI 高速采集并自动提取网页数据

本视频演示 PulsarRPAPro 的 AI 爬虫技术。

AI 爬虫指的是一组 AI 智能体,也就是 AI agents,它能够像真人一样网上冲浪,阅读理解在线网页,并且可以完全自动地分析网页,输出结构化数据或者知识图谱,从而支持各种各样的业务,譬如电商数据分析,或者大语言模型的检索增强生成(RAG)。

在网页上,凡人眼可见的数据,几乎都能够被完整、精确提取出来,譬如,标题、价格、折扣、优惠、配送等等关键字段。绝大多数字段,准确率和召回率均超过99%。

在传统上,我们需要使用 PulsarRPA,或者 selenium 这样的浏览器自动化工具,花费大量时间,编写X-SQL、CSSPath、XPath、正则表达式等,来提取网页数据,将网页转变成可以直接分析的结构化数据。

使用 platon.ai 的AI 爬虫 PulsarRPAPro,可以像真人一样无障碍网上冲浪,自动提取网页中的所有字段,输出结构化数据。

执行 PulsarRPAPro 后,系统将打开入口页面,和网页进行交互,等待延迟加载的网页内容也完整呈现。

在所有网页内容完整呈现后,PulsarRPAPro 智能地找到了所有商品链接,并逐一访问这些链接,浏览商品页面。

和真人不同的是,PulsarRPAPro 访问速度非常快,访问的网页数量没有限制,单机每天访问十万、几十万网页,采集数千万、上亿数据点,毫无压力。

访问所有网页后,PulsarRPAPro 直接告诉你,网页上所有的数据,并且保存为后续分析所需的格式。

网页理解

零干预将网站还原为数据

给定入口链接,柏拉图 AI 识别、浏览并解读最重要的链出页,输出全部字段:

      select * from harvest('https://www.amazon.com/b?node=3117954011');
     

AI 已浏览120个网页,已理解8组数据共142个字段。 以下显示第2组数据,该组数据共包含10个字段,对应网页区域 #centerCol

  URL Title By Brand Stars Ratings Bar QAs Fees Other Sellers Price
1 Amazon.com: BLACK+DECKER 6 quart 11-in-1 Cooking Pot, Stainless Steel, Pressure Cooker, Slow Cooker, Multi-Cooker, PR100 BLACK+DECKER 6 quart 11-in-1 Cooking Pot, Stainless Steel, Pressure Cooker, Slow Cooker, Multi-Cooker, PR100 by BLACK+DECKER 4.2 out of 5 stars 129 ratings | 89 answered questions + No Import Fees Deposit & ¥40.72 Shipping to Hong Kong New (5) from ¥54.17
2 Amazon.com: BLACK+DECKER 6 quart 11-in-1 Cooking Pot, Stainless Steel, Pressure Cooker, Slow Cooker, Multi-Cooker, PR100 BLACK+DECKER 6 quart 11-in-1 Cooking Pot, Stainless Steel, Pressure Cooker, Slow Cooker, Multi-Cooker, PR100 by BLACK+DECKER 4.2 out of 5 stars 129 ratings | 89 answered questions + No Import Fees Deposit & ¥40.72 Shipping to Hong Kong New (5) from ¥54.17
3 Amazon.com: Crock Pot 6 Quart 8 in 1 Multi Use Express Crock Programmable Pressure Cooker, Slow Cooker, Sauté & Steamer | Stainless Steel (SCCPPC60... Crock Pot 6 Quart 8 in 1 Multi Use Express Crock Programmable Pressure Cooker, Slow Cooker, Sauté & Steamer | Stainless Steel (SCCPPC600 V1) by Crockpot 4.2 out of 5 stars 2,086 ratings | 670 answered questions There is a newer model of this item: New (31) from ¥74.79
4 Amazon.com: Crockpot Thermoshield 6 Quart Manual Slow Cooker, Black Crockpot Thermoshield 6 Quart Manual Slow Cooker, Black by Crockpot 4.1 out of 5 stars 150 ratings | 47 answered questions + No Import Fees Deposit & ¥47.40 Shipping to Hong Kong New & Used (12) from ¥59.99
5 Amazon.com: GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT by GoWISE USA 3.9 out of 5 stars 927 ratings | 498 answered questions + No Import Fees Deposit & ¥70.96 Shipping to Hong Kong New & Used (4) from ¥113.18
6 Amazon.com: GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT by GoWISE USA 3.9 out of 5 stars 927 ratings | 498 answered questions + No Import Fees Deposit & ¥70.96 Shipping to Hong Kong New & Used (4) from ¥113.18
7 Amazon.com: GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT by GoWISE USA 3.9 out of 5 stars 927 ratings | 498 answered questions + No Import Fees Deposit & ¥70.96 Shipping to Hong Kong New & Used (4) from ¥113.18
8 Amazon.com: Gourmia GPC400 4 Qt Digital Multi-Mode SmartPot Pressure Cooker - 13 Cook Modes - Removable Pot - 24-Hour Delay Timer - Automatic Keep ... Gourmia GPC400 4 Qt Digital Multi-Mode SmartPot Pressure Cooker - 13 Cook Modes - Removable Pot - 24-Hour Delay Timer - Automatic Keep Warm - LCD Display - Pressure Sensor Lid Lock - Recipe Book by Gourmia 4.2 out of 5 stars 363 ratings | 171 answered questions + No Import Fees Deposit & ¥31.80 Shipping to Hong Kong    
9 Amazon.com: Mealthy MultiPot 9-in-1 Programmable Pressure Cooker 6 Quarts with Stainless Steel Pot, Steamer Basket, instant access to recipe app. P... Mealthy MultiPot 9-in-1 Programmable Pressure Cooker 6 Quarts with Stainless Steel Pot, Steamer Basket, instant access to recipe app. Pressure cook, slow cook, sauté, rice cooker, yogurt, steam by Mealthy 4.7 out of 5 stars 1,593 ratings | 934 answered questions   New & Used (3) from ¥169.99
10 Amazon.com: Ninja Instant, 1000-Watt Pressure, Slow, Multi Cooker, and Steamer with 6-Quart Ceramic Coated Pot & Steam Rack (PC101), Si, Black/Silver Ninja Instant, 1000-Watt Pressure, Slow, Multi Cooker, and Steamer with 6-Quart Ceramic Coated Pot & Steam Rack (PC101), Si, Black/Silver by Ninja 4.7 out of 5 stars 120 ratings | 65 answered questions This product is available as Renewed. New & Used (11) from ¥54.95
11 Amazon.com: Power Pressure Cooker XL 10 Qt Power Pressure Cooker XL 10 Qt by Power Pressure Cooker XL 4.1 out of 5 stars 2,977 ratings | 1000+ answered questions + No Import Fees Deposit & ¥51.68 Shipping to Hong Kong New & Used (6) from ¥159.00
12 Amazon.com: Presto 02141 6-Quart Electric Pressure Cooker, Stainless, Black, Silver Presto 02141 6-Quart Electric Pressure Cooker, Stainless, Black, Silver by Presto 4.2 out of 5 stars 54 ratings | 17 answered questions + No Import Fees Deposit & ¥38.45 Shipping to Hong Kong New & Used (33) from ¥59.99
执行 AI 自动挖掘 执行 AI 辅助挖掘