Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.126805288s FAIL (2025-07-21 01:20:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.126833069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.849297647s FAIL (2025-07-21 01:19:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.84932554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 7.815417935s FAIL (2025-07-21 01:18:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.815442992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.148574368s FAIL (2025-07-21 01:17:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.148603014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.902966934s FAIL (2025-07-21 01:17:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.902992944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.905108844s FAIL (2025-07-21 01:16:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.905142718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.795954118s FAIL (2025-07-21 01:16:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.795979858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.326499041s FAIL (2025-07-21 01:15:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.326527335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.689451882s FAIL (2025-07-21 01:14:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.68947752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.651319688s FAIL (2025-07-21 01:13:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.651343754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.836516157s FAIL (2025-07-21 01:12:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.836542959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.024818868s FAIL (2025-07-21 01:11:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.02484089s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.715220409s FAIL (2025-07-21 01:11:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.715243373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.847569625s FAIL (2025-07-21 01:10:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.847597298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.799615799s FAIL (2025-07-21 01:10:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.799644584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.882859114s FAIL (2025-07-21 01:08:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.882881167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.875499529s FAIL (2025-07-21 01:08:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.875529285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.42213324s FAIL (2025-07-21 01:07:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.422172224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.73579226s FAIL (2025-07-21 01:06:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.73581819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.839518379s FAIL (2025-07-21 01:06:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.839543606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 7.192261708s FAIL (2025-07-21 01:05:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.192299689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.932643346s FAIL (2025-07-21 01:04:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.932670119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.944277345s FAIL (2025-07-21 01:03:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.944301052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.744687629s FAIL (2025-07-21 01:03:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.744716844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.088940971s FAIL (2025-07-21 01:03:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.088969725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.076836561s FAIL (2025-07-21 01:02:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.076860847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.730314919s FAIL (2025-07-21 01:01:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.73034136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.020407116s FAIL (2025-07-21 01:00:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.020422665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.197235035s FAIL (2025-07-21 01:00:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.197257618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.037770944s FAIL (2025-07-21 00:59:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.037802805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.79747512s FAIL (2025-07-21 00:59:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.797509204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.856031467s FAIL (2025-07-21 00:58:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.856060944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.994779356s FAIL (2025-07-21 00:57:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.99480704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.217055716s FAIL (2025-07-21 00:56:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.217085532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.242690222s FAIL (2025-07-21 00:56:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments... Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil> Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments... Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment... Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment. Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece... Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments... Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970 , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295 , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082 , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877 , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772 , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842 , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589 , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154 , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119 , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263 , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010 , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil> Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806 , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765 , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875 , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048 , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments... Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.242718856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.858755487s FAIL (2025-07-20 21:57:12 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.858777778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.907159294s FAIL (2025-07-20 21:56:19 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.907182149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.858831042s FAIL (2025-07-20 21:55:20 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.858878442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.847938793s FAIL (2025-07-20 21:54:39 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.847961265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.892890235s FAIL (2025-07-20 21:53:27 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.892915714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.856109463s FAIL (2025-07-20 21:52:16 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.856130283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.857891454s FAIL (2025-07-20 21:51:09 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.857913086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.928398379s FAIL (2025-07-20 21:50:29 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.928434607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.872083509s FAIL (2025-07-20 21:49:35 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.872109088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.88985904s FAIL (2025-07-20 21:48:29 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.88991681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.929025545s FAIL (2025-07-20 21:47:48 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.929070001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.941910923s FAIL (2025-07-20 21:46:38 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.94195715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.911393141s FAIL (2025-07-20 21:46:21 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.911417437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.871399369s FAIL (2025-07-20 21:45:56 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 1.871433513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.06088303s FAIL (2025-07-20 21:44:47 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== ad=36912]: Done uploading segments... Jul 20 21:04:13.677519035 (46h13m20.05009839s) [upload=36912]: Upload finished. err: <nil> Jul 20 21:04:38.321894955 (46h13m44.69447431s) [upload=36917]: Starting upload Jul 20 21:04:38.321948858 (46h13m44.694528213s) [upload=36917]: Done starting upload Jul 20 21:04:38.321998483 (46h13m44.694577838s) [upload=36917]: Uploading segments... Jul 20 21:04:38.322030634 (46h13m44.694609979s) [upload=36917]: Got next segment. Inline: true Jul 20 21:04:38.322037026 (46h13m44.694616371s) [upload=36917]: Waiting for error group managing segments... Jul 20 21:04:38.322068807 (46h13m44.694648162s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:04:38.322073266 (46h13m44.694652621s) [upload=36917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:04:38.322075671 (46h13m44.694655026s) [upload=36917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:04:38.620420428 (46h13m44.992999783s) [upload=36917]: Done uploading segments... Jul 20 21:04:38.620437882 (46h13m44.993017237s) [upload=36917]: Upload finished. err: <nil> Jul 20 21:05:12.846004647 (46h14m19.218584002s) [upload=36923]: Starting upload Jul 20 21:05:12.846059702 (46h14m19.218639057s) [upload=36923]: Done starting upload Jul 20 21:05:12.846111621 (46h14m19.218690976s) [upload=36923]: Uploading segments... Jul 20 21:05:12.846147460 (46h14m19.218726815s) [upload=36923]: Got next segment. Inline: true Jul 20 21:05:12.846151006 (46h14m19.218730351s) [upload=36923]: Waiting for error group managing segments... Jul 20 21:05:12.846182256 (46h14m19.218761611s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:05:12.846186936 (46h14m19.218766291s) [upload=36923]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:05:12.846189631 (46h14m19.218768986s) [upload=36923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:05:13.107295768 (46h14m19.479875123s) [upload=36923]: Done uploading segments... Jul 20 21:05:13.107315646 (46h14m19.479895001s) [upload=36923]: Upload finished. err: <nil> Jul 20 21:06:16.301788086 (46h15m22.674367441s) [upload=36939]: Starting upload Jul 20 21:06:16.301871996 (46h15m22.674451351s) [upload=36939]: Done starting upload Jul 20 21:06:16.301955386 (46h15m22.674534741s) [upload=36939]: Uploading segments... Jul 20 21:06:16.302046020 (46h15m22.674625375s) [upload=36939]: Got next segment. Inline: true Jul 20 21:06:16.302050829 (46h15m22.674630174s) [upload=36939]: Waiting for error group managing segments... Jul 20 21:06:16.302093029 (46h15m22.674672384s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:06:16.302098530 (46h15m22.674677885s) [upload=36939]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:06:16.302100815 (46h15m22.67468017s) [upload=36939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:06:16.574020181 (46h15m22.946599536s) [upload=36939]: Done uploading segments... Jul 20 21:06:16.574039678 (46h15m22.946619033s) [upload=36939]: Upload finished. err: <nil> Jul 20 21:07:05.296385791 (46h16m11.668965147s) [upload=36948]: Starting upload Jul 20 21:07:05.296439294 (46h16m11.669018649s) [upload=36948]: Done starting upload Jul 20 21:07:05.296491223 (46h16m11.669070578s) [upload=36948]: Uploading segments... Jul 20 21:07:05.296525328 (46h16m11.669104683s) [upload=36948]: Got next segment. Inline: true Jul 20 21:07:05.296529025 (46h16m11.66910838s) [upload=36948]: Waiting for error group managing segments... Jul 20 21:07:05.296565796 (46h16m11.669145151s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:05.296570906 (46h16m11.669150251s) [upload=36948]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:05.296573851 (46h16m11.669153196s) [upload=36948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:05.573631929 (46h16m11.946211284s) [upload=36948]: Done uploading segments... Jul 20 21:07:05.573649023 (46h16m11.946228368s) [upload=36948]: Upload finished. err: <nil> Jul 20 21:07:56.373712750 (46h17m2.746292105s) [upload=36960]: Starting upload Jul 20 21:07:56.373783155 (46h17m2.74636251s) [upload=36960]: Done starting upload Jul 20 21:07:56.373851666 (46h17m2.746431011s) [upload=36960]: Uploading segments... Jul 20 21:07:56.373949193 (46h17m2.746528548s) [upload=36960]: Got next segment. Inline: true Jul 20 21:07:56.373953731 (46h17m2.746533086s) [upload=36960]: Waiting for error group managing segments... Jul 20 21:07:56.374014357 (46h17m2.746593712s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:07:56.374020318 (46h17m2.746599673s) [upload=36960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:07:56.374025790 (46h17m2.746605155s) [upload=36960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:07:56.652828505 (46h17m3.02540786s) [upload=36960]: Done uploading segments... Jul 20 21:07:56.652845779 (46h17m3.025425134s) [upload=36960]: Upload finished. err: <nil> Jul 20 21:09:02.925460116 (46h18m9.298039461s) [upload=36974]: Starting upload Jul 20 21:09:02.925517746 (46h18m9.298097101s) [upload=36974]: Done starting upload Jul 20 21:09:02.925566349 (46h18m9.298145704s) [upload=36974]: Uploading segments... Jul 20 21:09:02.925612427 (46h18m9.298191782s) [upload=36974]: Got next segment. Inline: true Jul 20 21:09:02.925617185 (46h18m9.298196531s) [upload=36974]: Waiting for error group managing segments... Jul 20 21:09:02.925653295 (46h18m9.29823265s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:02.925658585 (46h18m9.29823794s) [upload=36974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:02.925661090 (46h18m9.298240445s) [upload=36974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:03.198427337 (46h18m9.571006692s) [upload=36974]: Done uploading segments... Jul 20 21:09:03.198446344 (46h18m9.571025699s) [upload=36974]: Upload finished. err: <nil> Jul 20 21:09:50.518152044 (46h18m56.890731399s) [upload=36989]: Starting upload Jul 20 21:09:50.518213932 (46h18m56.890793287s) [upload=36989]: Done starting upload Jul 20 21:09:50.518271142 (46h18m56.890850497s) [upload=36989]: Uploading segments... Jul 20 21:09:50.518299306 (46h18m56.890878661s) [upload=36989]: Got next segment. Inline: true Jul 20 21:09:50.518302833 (46h18m56.890882188s) [upload=36989]: Waiting for error group managing segments... Jul 20 21:09:50.518337399 (46h18m56.890916754s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:09:50.518342077 (46h18m56.890921432s) [upload=36989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:09:50.518346145 (46h18m56.8909255s) [upload=36989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:09:50.793305337 (46h18m57.165884692s) [upload=36989]: Done uploading segments... Jul 20 21:09:50.793323522 (46h18m57.165902867s) [upload=36989]: Upload finished. err: <nil> Jul 20 21:10:06.910915760 (46h19m13.283495155s) [upload=36992]: Starting upload Jul 20 21:10:06.911047172 (46h19m13.283626547s) [upload=36992]: Done starting upload Jul 20 21:10:06.911137204 (46h19m13.283716559s) [upload=36992]: Uploading segments... Jul 20 21:10:06.911222206 (46h19m13.283801561s) [upload=36992]: Got next segment. Inline: true Jul 20 21:10:06.911230974 (46h19m13.283810329s) [upload=36992]: Waiting for error group managing segments... Jul 20 21:10:06.911275638 (46h19m13.283854994s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:06.911280938 (46h19m13.283860304s) [upload=36992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:06.911284506 (46h19m13.283863861s) [upload=36992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:07.188023904 (46h19m13.560603259s) [upload=36992]: Done uploading segments... Jul 20 21:10:07.188046277 (46h19m13.560625642s) [upload=36992]: Upload finished. err: <nil> Jul 20 21:10:24.474001527 (46h19m30.846580882s) [upload=36997]: Starting upload Jul 20 21:10:24.474105496 (46h19m30.846684891s) [upload=36997]: Done starting upload Jul 20 21:10:24.474177664 (46h19m30.846757019s) [upload=36997]: Uploading segments... Jul 20 21:10:24.474217580 (46h19m30.846796935s) [upload=36997]: Got next segment. Inline: true Jul 20 21:10:24.474223562 (46h19m30.846802907s) [upload=36997]: Waiting for error group managing segments... Jul 20 21:10:24.474274910 (46h19m30.846854255s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:10:24.474281603 (46h19m30.846860968s) [upload=36997]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:10:24.474287164 (46h19m30.846866529s) [upload=36997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:10:24.742102808 (46h19m31.114682163s) [upload=36997]: Done uploading segments... Jul 20 21:10:24.742122797 (46h19m31.114702152s) [upload=36997]: Upload finished. err: <nil> Jul 20 21:11:24.345418141 (46h20m30.717997506s) [upload=37010]: Starting upload Jul 20 21:11:24.345503054 (46h20m30.718082419s) [upload=37010]: Done starting upload Jul 20 21:11:24.345563129 (46h20m30.718142484s) [upload=37010]: Uploading segments... Jul 20 21:11:24.345600551 (46h20m30.718179906s) [upload=37010]: Got next segment. Inline: true Jul 20 21:11:24.345605720 (46h20m30.718185075s) [upload=37010]: Waiting for error group managing segments... Jul 20 21:11:24.345648973 (46h20m30.718228328s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 20 21:11:24.345656998 (46h20m30.718236354s) [upload=37010]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Jul 20 21:11:24.345659954 (46h20m30.718239309s) [upload=37010]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 20 21:11:24.644900284 (46h20m31.017479639s) [upload=37010]: Done uploading segments... Jul 20 21:11:24.644918277 (46h20m31.017497622s) [upload=37010]: Upload finished. err: <nil> Duration: 2.060908789s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)