本视频介绍如何训练机器学习模型,高精度提取网页数据,一次训练,永久有效。
具体方法是:使用PulsarRPAPro采集商品详情页,使用无监督学习技术自动提取数据,得到一个初始的数据集,经过简单的人工校验和数据标注之后,形成了一个可以训练机器学习模型的数据集。用这个数据集训练机器学习模型,就可以高精度提取网页数据。
举个例子,我们要采集amazon上的产品数据,譬如标题,评分,价格等等。在传统上,我们需要使用PulsarRPA,selenium这样的浏览器自动化工具,逐一打开商品详情页,花费大量时间,编写X-SQL,正则表达式等来提取网页数据,将网页转变成可直接分析的结构化数据。
本视频中,我们使用 PulsarRPAPro 的无监督学习技术自动将每一个字段提取出来,形成初步的数据集,对自动提取出来的字段进行人工校验,剔除错误数据,给每个字段一个名字,得到一个良好标注的数据集,训练一个机器学习模型。
经过训练后,整体准确率超过 98%,而绝大部分字段的准确率和召回率均达到了100%。随着数据质量的提升和训练集的扩大,精度将继续提升。
最后,我们将模型预测结果导出,也就是网页数据提取结果。
使用无监督学习+监督学习进行网页数据提取,我们将网页数据提取的人效提升了1000倍以上,提升了数据提取准确率,降低了人员技能要求,同时也不再需要频繁维护数据提取规则。
在下一个视频中,我们将介绍,采集多个站点网页,一次标注,训练机器学习模型,从而实现单一机器学习模型,自动提取多站点数据。
零干预将网站还原为数据
给定入口链接,柏拉图 AI 识别、浏览并解读最重要的链出页,输出全部字段:
select * from harvest('https://www.amazon.com/b?node=3117954011');
AI 已浏览120个网页,已理解8组数据共142个字段。 以下显示第2组数据,该组数据共包含10个字段,对应网页区域 #centerCol
URL | Title | By | Brand | Stars | Ratings | Bar | QAs | Fees | Other Sellers | Price | ||
1 | Amazon.com: BLACK+DECKER 6 quart 11-in-1 Cooking Pot, Stainless Steel, Pressure Cooker, Slow Cooker, Multi-Cooker, PR100 | BLACK+DECKER 6 quart 11-in-1 Cooking Pot, Stainless Steel, Pressure Cooker, Slow Cooker, Multi-Cooker, PR100 | by | BLACK+DECKER | 4.2 out of 5 stars | 129 ratings | | | 89 answered questions | + No Import Fees Deposit & ¥40.72 Shipping to Hong Kong | New (5) from | ¥54.17 | |
2 | Amazon.com: BLACK+DECKER 6 quart 11-in-1 Cooking Pot, Stainless Steel, Pressure Cooker, Slow Cooker, Multi-Cooker, PR100 | BLACK+DECKER 6 quart 11-in-1 Cooking Pot, Stainless Steel, Pressure Cooker, Slow Cooker, Multi-Cooker, PR100 | by | BLACK+DECKER | 4.2 out of 5 stars | 129 ratings | | | 89 answered questions | + No Import Fees Deposit & ¥40.72 Shipping to Hong Kong | New (5) from | ¥54.17 | |
3 | Amazon.com: Crock Pot 6 Quart 8 in 1 Multi Use Express Crock Programmable Pressure Cooker, Slow Cooker, Sauté & Steamer | Stainless Steel (SCCPPC60... | Crock Pot 6 Quart 8 in 1 Multi Use Express Crock Programmable Pressure Cooker, Slow Cooker, Sauté & Steamer | Stainless Steel (SCCPPC600 V1) | by | Crockpot | 4.2 out of 5 stars | 2,086 ratings | | | 670 answered questions | There is a newer model of this item: | New (31) from | ¥74.79 | |
4 | Amazon.com: Crockpot Thermoshield 6 Quart Manual Slow Cooker, Black | Crockpot Thermoshield 6 Quart Manual Slow Cooker, Black | by | Crockpot | 4.1 out of 5 stars | 150 ratings | | | 47 answered questions | + No Import Fees Deposit & ¥47.40 Shipping to Hong Kong | New & Used (12) from | ¥59.99 | |
5 | Amazon.com: GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT | GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT | by | GoWISE USA | 3.9 out of 5 stars | 927 ratings | | | 498 answered questions | + No Import Fees Deposit & ¥70.96 Shipping to Hong Kong | New & Used (4) from | ¥113.18 | |
6 | Amazon.com: GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT | GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT | by | GoWISE USA | 3.9 out of 5 stars | 927 ratings | | | 498 answered questions | + No Import Fees Deposit & ¥70.96 Shipping to Hong Kong | New & Used (4) from | ¥113.18 | |
7 | Amazon.com: GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT | GoWISE USA GW22637 4th-Generation Electric Pressure Cooker with rice scooper, and measuring cup, 14 QT | by | GoWISE USA | 3.9 out of 5 stars | 927 ratings | | | 498 answered questions | + No Import Fees Deposit & ¥70.96 Shipping to Hong Kong | New & Used (4) from | ¥113.18 | |
8 | Amazon.com: Gourmia GPC400 4 Qt Digital Multi-Mode SmartPot Pressure Cooker - 13 Cook Modes - Removable Pot - 24-Hour Delay Timer - Automatic Keep ... | Gourmia GPC400 4 Qt Digital Multi-Mode SmartPot Pressure Cooker - 13 Cook Modes - Removable Pot - 24-Hour Delay Timer - Automatic Keep Warm - LCD Display - Pressure Sensor Lid Lock - Recipe Book | by | Gourmia | 4.2 out of 5 stars | 363 ratings | | | 171 answered questions | + No Import Fees Deposit & ¥31.80 Shipping to Hong Kong | |||
9 | Amazon.com: Mealthy MultiPot 9-in-1 Programmable Pressure Cooker 6 Quarts with Stainless Steel Pot, Steamer Basket, instant access to recipe app. P... | Mealthy MultiPot 9-in-1 Programmable Pressure Cooker 6 Quarts with Stainless Steel Pot, Steamer Basket, instant access to recipe app. Pressure cook, slow cook, sauté, rice cooker, yogurt, steam | by | Mealthy | 4.7 out of 5 stars | 1,593 ratings | | | 934 answered questions | New & Used (3) from | ¥169.99 | ||
10 | Amazon.com: Ninja Instant, 1000-Watt Pressure, Slow, Multi Cooker, and Steamer with 6-Quart Ceramic Coated Pot & Steam Rack (PC101), Si, Black/Silver | Ninja Instant, 1000-Watt Pressure, Slow, Multi Cooker, and Steamer with 6-Quart Ceramic Coated Pot & Steam Rack (PC101), Si, Black/Silver | by | Ninja | 4.7 out of 5 stars | 120 ratings | | | 65 answered questions | This product is available as Renewed. | New & Used (11) from | ¥54.95 | |
11 | Amazon.com: Power Pressure Cooker XL 10 Qt | Power Pressure Cooker XL 10 Qt | by | Power Pressure Cooker XL | 4.1 out of 5 stars | 2,977 ratings | | | 1000+ answered questions | + No Import Fees Deposit & ¥51.68 Shipping to Hong Kong | New & Used (6) from | ¥159.00 | |
12 | Amazon.com: Presto 02141 6-Quart Electric Pressure Cooker, Stainless, Black, Silver | Presto 02141 6-Quart Electric Pressure Cooker, Stainless, Black, Silver | by | Presto | 4.2 out of 5 stars | 54 ratings | | | 17 answered questions | + No Import Fees Deposit & ¥38.45 Shipping to Hong Kong | New & Used (33) from | ¥59.99 |