Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X..XX....] duration: 5.776032874s trace: https://jaeger.dev.storj.io/trace/1e9192a7c380565f FAIL (2025-12-31 15:43:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA) context deadline exceeded ======LOGS====== Dec 31 15:40:02.183200486 (624h11m2.054283531s) [upload=546961]: Starting upload Dec 31 15:40:02.183289697 (624h11m2.054372742s) [upload=546961]: Done starting upload Dec 31 15:40:02.183557467 (624h11m2.054640512s) [upload=546961]: Uploading segments... Dec 31 15:40:02.183626839 (624h11m2.054709924s) [upload=546961]: Got next segment. Inline: false Dec 31 15:40:02.183683035 (624h11m2.05476608s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:40:02.183691231 (624h11m2.054774276s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:40:02.433064484 (624h11m2.304147539s) [seg_pos={0 0}, upload=546961]: Begin upload segment... Dec 31 15:40:02.433122744 (624h11m2.304205799s) [seg_pos={0 0}, upload=546961]: Done begin upload segment. Dec 31 15:40:02.433159664 (624h11m2.304242719s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.434864581 (624h11m2.305947656s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.435672529 (624h11m2.306755574s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436304893 (624h11m2.307387938s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436829121 (624h11m2.307912166s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.438953957 (624h11m2.310037002s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.440128664 (624h11m2.311211709s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.445437173 (624h11m2.316520208s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.447396233 (624h11m2.318479288s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.448528338 (624h11m2.319611393s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.449941790 (624h11m2.321024855s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.452078879 (624h11m2.323161924s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.455900075 (624h11m2.32698315s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458138027 (624h11m2.329221072s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458601058 (624h11m2.329684103s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.460663175 (624h11m2.33174622s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:04.142907034 (624h11m4.013990079s) [upload=546961]: Next returned nil segment Dec 31 15:40:04.142928124 (624h11m4.014011169s) [upload=546961]: Waiting for error group managing segments... Dec 31 15:40:04.487357616 (624h11m4.358440661s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.505736883 (624h11m4.376819928s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.509642600 (624h11m4.380725645s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.511026896 (624h11m4.382109941s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516140091 (624h11m4.387223146s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516207069 (624h11m4.387290124s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.517863183 (624h11m4.388946228s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.554784937 (624h11m4.425867992s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.571180697 (624h11m4.442263732s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.593032015 (624h11m4.46411506s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.803506067 (624h11m4.674589112s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.818445714 (624h11m4.689528759s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856531133 (624h11m4.727614178s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856558596 (624h11m4.727641651s) [seg_pos={0 0}, upload=546961]: Segment reached optimal threshold of 13 pieces. Dec 31 15:40:04.856807299 (624h11m4.727890344s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.856819383 (624h11m4.727902418s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857304316 (624h11m4.728387372s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857313794 (624h11m4.728396839s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857472977 (624h11m4.728556032s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857481664 (624h11m4.728564709s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857504938 (624h11m4.728587983s) [seg_pos={0 0}, upload=546961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:40:04.857638522 (624h11m4.728721577s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:40:04.857646417 (624h11m4.728729462s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:40:05.159347058 (624h11m5.030430103s) [upload=546961]: Done uploading segments... Dec 31 15:40:05.159476614 (624h11m5.030559659s) [upload=546961]: Upload finished. err: <nil> Duration: 5.776061357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [....X..XX...........XXX..X.XX..XX..XXXXX...X.X.X.......X.XX.] duration: 4.598596002s trace: https://jaeger.dev.storj.io/trace/3426de3f9a9a50f6 FAIL (2025-12-31 15:42:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.598623043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X..XX...] duration: 6.914225696s trace: https://jaeger.dev.storj.io/trace/1e9192a7c380565f FAIL (2025-12-31 15:42:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA) context deadline exceeded ======LOGS====== Dec 31 15:40:02.183200486 (624h11m2.054283531s) [upload=546961]: Starting upload Dec 31 15:40:02.183289697 (624h11m2.054372742s) [upload=546961]: Done starting upload Dec 31 15:40:02.183557467 (624h11m2.054640512s) [upload=546961]: Uploading segments... Dec 31 15:40:02.183626839 (624h11m2.054709924s) [upload=546961]: Got next segment. Inline: false Dec 31 15:40:02.183683035 (624h11m2.05476608s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:40:02.183691231 (624h11m2.054774276s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:40:02.433064484 (624h11m2.304147539s) [seg_pos={0 0}, upload=546961]: Begin upload segment... Dec 31 15:40:02.433122744 (624h11m2.304205799s) [seg_pos={0 0}, upload=546961]: Done begin upload segment. Dec 31 15:40:02.433159664 (624h11m2.304242719s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.434864581 (624h11m2.305947656s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.435672529 (624h11m2.306755574s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436304893 (624h11m2.307387938s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436829121 (624h11m2.307912166s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.438953957 (624h11m2.310037002s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.440128664 (624h11m2.311211709s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.445437173 (624h11m2.316520208s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.447396233 (624h11m2.318479288s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.448528338 (624h11m2.319611393s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.449941790 (624h11m2.321024855s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.452078879 (624h11m2.323161924s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.455900075 (624h11m2.32698315s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458138027 (624h11m2.329221072s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458601058 (624h11m2.329684103s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.460663175 (624h11m2.33174622s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:04.142907034 (624h11m4.013990079s) [upload=546961]: Next returned nil segment Dec 31 15:40:04.142928124 (624h11m4.014011169s) [upload=546961]: Waiting for error group managing segments... Dec 31 15:40:04.487357616 (624h11m4.358440661s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.505736883 (624h11m4.376819928s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.509642600 (624h11m4.380725645s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.511026896 (624h11m4.382109941s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516140091 (624h11m4.387223146s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516207069 (624h11m4.387290124s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.517863183 (624h11m4.388946228s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.554784937 (624h11m4.425867992s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.571180697 (624h11m4.442263732s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.593032015 (624h11m4.46411506s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.803506067 (624h11m4.674589112s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.818445714 (624h11m4.689528759s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856531133 (624h11m4.727614178s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856558596 (624h11m4.727641651s) [seg_pos={0 0}, upload=546961]: Segment reached optimal threshold of 13 pieces. Dec 31 15:40:04.856807299 (624h11m4.727890344s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.856819383 (624h11m4.727902418s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857304316 (624h11m4.728387372s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857313794 (624h11m4.728396839s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857472977 (624h11m4.728556032s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857481664 (624h11m4.728564709s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857504938 (624h11m4.728587983s) [seg_pos={0 0}, upload=546961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:40:04.857638522 (624h11m4.728721577s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:40:04.857646417 (624h11m4.728729462s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:40:05.159347058 (624h11m5.030430103s) [upload=546961]: Done uploading segments... Dec 31 15:40:05.159476614 (624h11m5.030559659s) [upload=546961]: Upload finished. err: <nil> Duration: 6.914252888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.X.........X..........................X.] duration: 46.182587575s trace: https://jaeger.dev.storj.io/trace/1871e827e14ff94e FAIL (2025-12-31 15:42:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 65s) [noise=true , address=145.79.227.29:4964 , piece_id=P5UDAXIBGZRNNKOBZWBKQUNHXX3DGK5ETO2EIHUAFY6KXOFUSFNQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.346922930 (624h11m40.218005965s) [noise=true , address=64.49.13.169:4706 , piece_id=W2I7DOJCVBAL7G37ZJMIZKY25T57OL5XSYWBB24PXY527LFFKTTA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.347268037 (624h11m40.218351072s) [noise=true , address=64.49.13.140:4505 , piece_id=6GADCK3XVRHYL27DKBUP7QYHCLSRJJUBZUKERPDANYUO6PY4NGDQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.347590842 (624h11m40.218673877s) [noise=true , address=145.79.227.35:4492 , piece_id=AV4HOCQI5736Q7PWU6CFAT5A7PFWIEHQSGTIPQXXMNQ5ZWBLDWAQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.347910220 (624h11m40.218993265s) [noise=true , address=145.79.227.34:4933 , piece_id=AZKTYLPVTIZC7HH3PLRFYVCZFSNUGRWIXN7H622AYXQVNV4KBP4A, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.348219339 (624h11m40.219302384s) [noise=true , address=77.90.31.15:4716 , piece_id=KJMRZUE2W2XVV2IM4S3OV4JLB7VGVUBG5D7BEKC6FM2FF5RQB5NQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.350000450 (624h11m40.221083485s) [noise=true , address=145.79.227.30:4766 , piece_id=BYKZBS2ELMFKFWA4Z6L3PHGFL4G6I3AYRBKTTVERX4CTACO775HA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.351275458 (624h11m40.222358503s) [noise=true , address=145.79.227.12:4793 , piece_id=KFUN4V26YQMIF3JXM5DXQVA4T6G75AJUZRGCLEKWR2H4YJDUO3RQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.352475092 (624h11m40.223558127s) [noise=true , address=145.79.227.49:4347 , piece_id=OI452GNCW3WMOT2YPJIPNYJKWX6LVXJSCYSBVE67DKF6GBODVWJA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.354333701 (624h11m40.225416736s) [noise=true , address=77.90.31.17:4787 , piece_id=G6UX3HEWGCDHLSRMBQVZFFM3E7EI5YBZKWAUR2DR5XZ7XP7ZAI6A, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.355947705 (624h11m40.22703076s) [noise=true , address=145.79.227.40:4073 , piece_id=JJ3RQAQLTNJ6PDDGDBBHWLXDOW76N2P24GVYLYBCBJKM24PYQC5A, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.357594841 (624h11m40.228677876s) [noise=true , address=77.90.31.21:4889 , piece_id=6JT5XL532AAPY5P3RGMANISJ6LCFUX6L2KOUF5ZSRRJ2M4GOEIXQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.359228191 (624h11m40.230311226s) [noise=true , address=45.91.116.10:4087 , piece_id=XRWODBKLQ4KKO24UZ3NADWUU5HGULECSRDKY6GGHYRDYU3PVXYEA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.361937150 (624h11m40.233020195s) [noise=true , address=145.79.227.33:4888 , piece_id=V7QLO4EFH4EYZBJGHVQ54SCXH26NPSMHIN33DBVOCZNOLGLOUJHQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.365093810 (624h11m40.236176845s) [noise=true , address=145.79.227.13:4826 , piece_id=7S3VGSRZSKGELG25UDLKZDDTWYQ3A6B7UH2NRWEZLYWOVBT7Q5JQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.365504863 (624h11m40.236587898s) [noise=true , address=145.79.227.43:4180 , piece_id=3ISNUTTGKUMY6IJIOQUCLEG3FFPP7O3QV2TEXKQ3UILYZWPFFBLA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:41.755708010 (624h11m41.626791056s) [upload=546972]: Next returned nil segment Dec 31 15:40:41.755733780 (624h11m41.626816855s) [upload=546972]: Waiting for error group managing segments... Dec 31 15:40:42.116142698 (624h11m41.987225733s) [noise=true , address=64.49.13.140:4505 , piece_id=6GADCK3XVRHYL27DKBUP7QYHCLSRJJUBZUKERPDANYUO6PY4NGDQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.219781030 (624h11m42.090864065s) [noise=true , address=145.79.227.29:4964 , piece_id=P5UDAXIBGZRNNKOBZWBKQUNHXX3DGK5ETO2EIHUAFY6KXOFUSFNQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.229425325 (624h11m42.10050837s) [noise=true , address=145.79.227.7:4310 , piece_id=USIYY5LXTLO4UZJGVYPHAIMI6TTHLJXCJXXQ5UOLFQCWC4AWERWA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.244021186 (624h11m42.115104231s) [noise=true , address=145.79.227.12:4793 , piece_id=KFUN4V26YQMIF3JXM5DXQVA4T6G75AJUZRGCLEKWR2H4YJDUO3RQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.256656747 (624h11m42.127739782s) [noise=true , address=145.79.227.13:4826 , piece_id=7S3VGSRZSKGELG25UDLKZDDTWYQ3A6B7UH2NRWEZLYWOVBT7Q5JQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.359805887 (624h11m42.230888932s) [noise=true , address=77.90.31.10:4550 , piece_id=XBYR3TDIF6SQD5TAWO55AH2G3VGSPXACKESZ3TTCDJ7MYO3IRZ4Q, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.459294287 (624h11m42.330377333s) [noise=true , address=77.90.31.15:4716 , piece_id=KJMRZUE2W2XVV2IM4S3OV4JLB7VGVUBG5D7BEKC6FM2FF5RQB5NQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.532321942 (624h11m42.403404977s) [noise=true , address=45.91.116.10:4087 , piece_id=XRWODBKLQ4KKO24UZ3NADWUU5HGULECSRDKY6GGHYRDYU3PVXYEA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.123963673 (624h11m42.995046708s) [noise=true , address=145.79.227.30:4766 , piece_id=BYKZBS2ELMFKFWA4Z6L3PHGFL4G6I3AYRBKTTVERX4CTACO775HA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.159155373 (624h11m43.030238408s) [noise=true , address=77.90.31.17:4787 , piece_id=G6UX3HEWGCDHLSRMBQVZFFM3E7EI5YBZKWAUR2DR5XZ7XP7ZAI6A, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.164032570 (624h11m43.035115615s) [noise=true , address=145.79.227.34:4933 , piece_id=AZKTYLPVTIZC7HH3PLRFYVCZFSNUGRWIXN7H622AYXQVNV4KBP4A, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.177379765 (624h11m43.0484628s) [noise=true , address=145.79.227.40:4073 , piece_id=JJ3RQAQLTNJ6PDDGDBBHWLXDOW76N2P24GVYLYBCBJKM24PYQC5A, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.196808540 (624h11m43.067891575s) [noise=true , address=145.79.227.49:4347 , piece_id=OI452GNCW3WMOT2YPJIPNYJKWX6LVXJSCYSBVE67DKF6GBODVWJA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.278077838 (624h11m43.149160883s) [noise=true , address=145.79.227.33:4888 , piece_id=V7QLO4EFH4EYZBJGHVQ54SCXH26NPSMHIN33DBVOCZNOLGLOUJHQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.278101924 (624h11m43.149184969s) [seg_pos={0 0}, upload=546972]: Segment reached optimal threshold of 14 pieces. Dec 31 15:40:43.278350387 (624h11m43.149433442s) [noise=true , address=145.79.227.35:4492 , piece_id=AV4HOCQI5736Q7PWU6CFAT5A7PFWIEHQSGTIPQXXMNQ5ZWBLDWAQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:43.278359895 (624h11m43.14944293s) [noise=true , address=145.79.227.35:4492 , piece_id=AV4HOCQI5736Q7PWU6CFAT5A7PFWIEHQSGTIPQXXMNQ5ZWBLDWAQ, seg_pos={0 0}, upload=546972]: Long tail cancellation due to optimal threshold Dec 31 15:40:43.278527825 (624h11m43.14961088s) [noise=true , address=145.79.227.43:4180 , piece_id=3ISNUTTGKUMY6IJIOQUCLEG3FFPP7O3QV2TEXKQ3UILYZWPFFBLA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:43.278537703 (624h11m43.149620738s) [noise=true , address=145.79.227.43:4180 , piece_id=3ISNUTTGKUMY6IJIOQUCLEG3FFPP7O3QV2TEXKQ3UILYZWPFFBLA, seg_pos={0 0}, upload=546972]: Long tail cancellation due to optimal threshold Dec 31 15:40:43.278567119 (624h11m43.149650174s) [noise=true , address=77.90.31.21:4889 , piece_id=6JT5XL532AAPY5P3RGMANISJ6LCFUX6L2KOUF5ZSRRJ2M4GOEIXQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:43.278586106 (624h11m43.149669151s) [noise=true , address=77.90.31.21:4889 , piece_id=6JT5XL532AAPY5P3RGMANISJ6LCFUX6L2KOUF5ZSRRJ2M4GOEIXQ, seg_pos={0 0}, upload=546972]: Long tail cancellation due to optimal threshold Dec 31 15:40:43.278693090 (624h11m43.149776145s) [noise=true , address=64.49.13.169:4706 , piece_id=W2I7DOJCVBAL7G37ZJMIZKY25T57OL5XSYWBB24PXY527LFFKTTA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:39176->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:39176->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->64.49.13.169:4706: use of closed network connection Dec 31 15:40:43.278701535 (624h11m43.14978457s) [noise=true , address=64.49.13.169:4706 , piece_id=W2I7DOJCVBAL7G37ZJMIZKY25T57OL5XSYWBB24PXY527LFFKTTA, seg_pos={0 0}, upload=546972]: Long tail cancellation due to optimal threshold Dec 31 15:40:43.278813819 (624h11m43.149896864s) [seg_pos={0 0}, upload=546972]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:40:43.278915964 (624h11m43.149999009s) [upload=546972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:40:43.278921965 (624h11m43.15000501s) [upload=546972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:40:43.578211354 (624h11m43.449294399s) [upload=546972]: Done uploading segments... Dec 31 15:40:43.578242012 (624h11m43.449325047s) [upload=546972]: Upload finished. err: <nil> Duration: 46.182605829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.....X..XX...........XXX..X.XX..XX..XXXXX...X.X.X.......X.XX] duration: 5.00140815s trace: https://jaeger.dev.storj.io/trace/3426de3f9a9a50f6 FAIL (2025-12-31 15:41:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001429461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X..XX..] duration: 5.774389265s trace: https://jaeger.dev.storj.io/trace/1e9192a7c380565f FAIL (2025-12-31 15:41:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA) context deadline exceeded ======LOGS====== Dec 31 15:40:02.183200486 (624h11m2.054283531s) [upload=546961]: Starting upload Dec 31 15:40:02.183289697 (624h11m2.054372742s) [upload=546961]: Done starting upload Dec 31 15:40:02.183557467 (624h11m2.054640512s) [upload=546961]: Uploading segments... Dec 31 15:40:02.183626839 (624h11m2.054709924s) [upload=546961]: Got next segment. Inline: false Dec 31 15:40:02.183683035 (624h11m2.05476608s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:40:02.183691231 (624h11m2.054774276s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:40:02.433064484 (624h11m2.304147539s) [seg_pos={0 0}, upload=546961]: Begin upload segment... Dec 31 15:40:02.433122744 (624h11m2.304205799s) [seg_pos={0 0}, upload=546961]: Done begin upload segment. Dec 31 15:40:02.433159664 (624h11m2.304242719s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.434864581 (624h11m2.305947656s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.435672529 (624h11m2.306755574s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436304893 (624h11m2.307387938s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436829121 (624h11m2.307912166s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.438953957 (624h11m2.310037002s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.440128664 (624h11m2.311211709s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.445437173 (624h11m2.316520208s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.447396233 (624h11m2.318479288s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.448528338 (624h11m2.319611393s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.449941790 (624h11m2.321024855s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.452078879 (624h11m2.323161924s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.455900075 (624h11m2.32698315s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458138027 (624h11m2.329221072s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458601058 (624h11m2.329684103s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.460663175 (624h11m2.33174622s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:04.142907034 (624h11m4.013990079s) [upload=546961]: Next returned nil segment Dec 31 15:40:04.142928124 (624h11m4.014011169s) [upload=546961]: Waiting for error group managing segments... Dec 31 15:40:04.487357616 (624h11m4.358440661s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.505736883 (624h11m4.376819928s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.509642600 (624h11m4.380725645s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.511026896 (624h11m4.382109941s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516140091 (624h11m4.387223146s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516207069 (624h11m4.387290124s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.517863183 (624h11m4.388946228s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.554784937 (624h11m4.425867992s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.571180697 (624h11m4.442263732s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.593032015 (624h11m4.46411506s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.803506067 (624h11m4.674589112s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.818445714 (624h11m4.689528759s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856531133 (624h11m4.727614178s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856558596 (624h11m4.727641651s) [seg_pos={0 0}, upload=546961]: Segment reached optimal threshold of 13 pieces. Dec 31 15:40:04.856807299 (624h11m4.727890344s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.856819383 (624h11m4.727902418s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857304316 (624h11m4.728387372s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857313794 (624h11m4.728396839s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857472977 (624h11m4.728556032s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857481664 (624h11m4.728564709s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857504938 (624h11m4.728587983s) [seg_pos={0 0}, upload=546961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:40:04.857638522 (624h11m4.728721577s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:40:04.857646417 (624h11m4.728729462s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:40:05.159347058 (624h11m5.030430103s) [upload=546961]: Done uploading segments... Dec 31 15:40:05.159476614 (624h11m5.030559659s) [upload=546961]: Upload finished. err: <nil> Duration: 5.774416768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.X.........X..........................X] duration: 1m0.004988202s trace: https://jaeger.dev.storj.io/trace/1871e827e14ff94e FAIL (2025-12-31 15:41:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 65s) [noise=true , address=145.79.227.29:4964 , piece_id=P5UDAXIBGZRNNKOBZWBKQUNHXX3DGK5ETO2EIHUAFY6KXOFUSFNQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.346922930 (624h11m40.218005965s) [noise=true , address=64.49.13.169:4706 , piece_id=W2I7DOJCVBAL7G37ZJMIZKY25T57OL5XSYWBB24PXY527LFFKTTA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.347268037 (624h11m40.218351072s) [noise=true , address=64.49.13.140:4505 , piece_id=6GADCK3XVRHYL27DKBUP7QYHCLSRJJUBZUKERPDANYUO6PY4NGDQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.347590842 (624h11m40.218673877s) [noise=true , address=145.79.227.35:4492 , piece_id=AV4HOCQI5736Q7PWU6CFAT5A7PFWIEHQSGTIPQXXMNQ5ZWBLDWAQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.347910220 (624h11m40.218993265s) [noise=true , address=145.79.227.34:4933 , piece_id=AZKTYLPVTIZC7HH3PLRFYVCZFSNUGRWIXN7H622AYXQVNV4KBP4A, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.348219339 (624h11m40.219302384s) [noise=true , address=77.90.31.15:4716 , piece_id=KJMRZUE2W2XVV2IM4S3OV4JLB7VGVUBG5D7BEKC6FM2FF5RQB5NQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.350000450 (624h11m40.221083485s) [noise=true , address=145.79.227.30:4766 , piece_id=BYKZBS2ELMFKFWA4Z6L3PHGFL4G6I3AYRBKTTVERX4CTACO775HA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.351275458 (624h11m40.222358503s) [noise=true , address=145.79.227.12:4793 , piece_id=KFUN4V26YQMIF3JXM5DXQVA4T6G75AJUZRGCLEKWR2H4YJDUO3RQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.352475092 (624h11m40.223558127s) [noise=true , address=145.79.227.49:4347 , piece_id=OI452GNCW3WMOT2YPJIPNYJKWX6LVXJSCYSBVE67DKF6GBODVWJA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.354333701 (624h11m40.225416736s) [noise=true , address=77.90.31.17:4787 , piece_id=G6UX3HEWGCDHLSRMBQVZFFM3E7EI5YBZKWAUR2DR5XZ7XP7ZAI6A, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.355947705 (624h11m40.22703076s) [noise=true , address=145.79.227.40:4073 , piece_id=JJ3RQAQLTNJ6PDDGDBBHWLXDOW76N2P24GVYLYBCBJKM24PYQC5A, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.357594841 (624h11m40.228677876s) [noise=true , address=77.90.31.21:4889 , piece_id=6JT5XL532AAPY5P3RGMANISJ6LCFUX6L2KOUF5ZSRRJ2M4GOEIXQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.359228191 (624h11m40.230311226s) [noise=true , address=45.91.116.10:4087 , piece_id=XRWODBKLQ4KKO24UZ3NADWUU5HGULECSRDKY6GGHYRDYU3PVXYEA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.361937150 (624h11m40.233020195s) [noise=true , address=145.79.227.33:4888 , piece_id=V7QLO4EFH4EYZBJGHVQ54SCXH26NPSMHIN33DBVOCZNOLGLOUJHQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.365093810 (624h11m40.236176845s) [noise=true , address=145.79.227.13:4826 , piece_id=7S3VGSRZSKGELG25UDLKZDDTWYQ3A6B7UH2NRWEZLYWOVBT7Q5JQ, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:40.365504863 (624h11m40.236587898s) [noise=true , address=145.79.227.43:4180 , piece_id=3ISNUTTGKUMY6IJIOQUCLEG3FFPP7O3QV2TEXKQ3UILYZWPFFBLA, seg_pos={0 0}, upload=546972]: Uploading piece... Dec 31 15:40:41.755708010 (624h11m41.626791056s) [upload=546972]: Next returned nil segment Dec 31 15:40:41.755733780 (624h11m41.626816855s) [upload=546972]: Waiting for error group managing segments... Dec 31 15:40:42.116142698 (624h11m41.987225733s) [noise=true , address=64.49.13.140:4505 , piece_id=6GADCK3XVRHYL27DKBUP7QYHCLSRJJUBZUKERPDANYUO6PY4NGDQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.219781030 (624h11m42.090864065s) [noise=true , address=145.79.227.29:4964 , piece_id=P5UDAXIBGZRNNKOBZWBKQUNHXX3DGK5ETO2EIHUAFY6KXOFUSFNQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.229425325 (624h11m42.10050837s) [noise=true , address=145.79.227.7:4310 , piece_id=USIYY5LXTLO4UZJGVYPHAIMI6TTHLJXCJXXQ5UOLFQCWC4AWERWA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.244021186 (624h11m42.115104231s) [noise=true , address=145.79.227.12:4793 , piece_id=KFUN4V26YQMIF3JXM5DXQVA4T6G75AJUZRGCLEKWR2H4YJDUO3RQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.256656747 (624h11m42.127739782s) [noise=true , address=145.79.227.13:4826 , piece_id=7S3VGSRZSKGELG25UDLKZDDTWYQ3A6B7UH2NRWEZLYWOVBT7Q5JQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.359805887 (624h11m42.230888932s) [noise=true , address=77.90.31.10:4550 , piece_id=XBYR3TDIF6SQD5TAWO55AH2G3VGSPXACKESZ3TTCDJ7MYO3IRZ4Q, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.459294287 (624h11m42.330377333s) [noise=true , address=77.90.31.15:4716 , piece_id=KJMRZUE2W2XVV2IM4S3OV4JLB7VGVUBG5D7BEKC6FM2FF5RQB5NQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:42.532321942 (624h11m42.403404977s) [noise=true , address=45.91.116.10:4087 , piece_id=XRWODBKLQ4KKO24UZ3NADWUU5HGULECSRDKY6GGHYRDYU3PVXYEA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.123963673 (624h11m42.995046708s) [noise=true , address=145.79.227.30:4766 , piece_id=BYKZBS2ELMFKFWA4Z6L3PHGFL4G6I3AYRBKTTVERX4CTACO775HA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.159155373 (624h11m43.030238408s) [noise=true , address=77.90.31.17:4787 , piece_id=G6UX3HEWGCDHLSRMBQVZFFM3E7EI5YBZKWAUR2DR5XZ7XP7ZAI6A, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.164032570 (624h11m43.035115615s) [noise=true , address=145.79.227.34:4933 , piece_id=AZKTYLPVTIZC7HH3PLRFYVCZFSNUGRWIXN7H622AYXQVNV4KBP4A, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.177379765 (624h11m43.0484628s) [noise=true , address=145.79.227.40:4073 , piece_id=JJ3RQAQLTNJ6PDDGDBBHWLXDOW76N2P24GVYLYBCBJKM24PYQC5A, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.196808540 (624h11m43.067891575s) [noise=true , address=145.79.227.49:4347 , piece_id=OI452GNCW3WMOT2YPJIPNYJKWX6LVXJSCYSBVE67DKF6GBODVWJA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.278077838 (624h11m43.149160883s) [noise=true , address=145.79.227.33:4888 , piece_id=V7QLO4EFH4EYZBJGHVQ54SCXH26NPSMHIN33DBVOCZNOLGLOUJHQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: <nil> Dec 31 15:40:43.278101924 (624h11m43.149184969s) [seg_pos={0 0}, upload=546972]: Segment reached optimal threshold of 14 pieces. Dec 31 15:40:43.278350387 (624h11m43.149433442s) [noise=true , address=145.79.227.35:4492 , piece_id=AV4HOCQI5736Q7PWU6CFAT5A7PFWIEHQSGTIPQXXMNQ5ZWBLDWAQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:43.278359895 (624h11m43.14944293s) [noise=true , address=145.79.227.35:4492 , piece_id=AV4HOCQI5736Q7PWU6CFAT5A7PFWIEHQSGTIPQXXMNQ5ZWBLDWAQ, seg_pos={0 0}, upload=546972]: Long tail cancellation due to optimal threshold Dec 31 15:40:43.278527825 (624h11m43.14961088s) [noise=true , address=145.79.227.43:4180 , piece_id=3ISNUTTGKUMY6IJIOQUCLEG3FFPP7O3QV2TEXKQ3UILYZWPFFBLA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:43.278537703 (624h11m43.149620738s) [noise=true , address=145.79.227.43:4180 , piece_id=3ISNUTTGKUMY6IJIOQUCLEG3FFPP7O3QV2TEXKQ3UILYZWPFFBLA, seg_pos={0 0}, upload=546972]: Long tail cancellation due to optimal threshold Dec 31 15:40:43.278567119 (624h11m43.149650174s) [noise=true , address=77.90.31.21:4889 , piece_id=6JT5XL532AAPY5P3RGMANISJ6LCFUX6L2KOUF5ZSRRJ2M4GOEIXQ, seg_pos={0 0}, upload=546972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:43.278586106 (624h11m43.149669151s) [noise=true , address=77.90.31.21:4889 , piece_id=6JT5XL532AAPY5P3RGMANISJ6LCFUX6L2KOUF5ZSRRJ2M4GOEIXQ, seg_pos={0 0}, upload=546972]: Long tail cancellation due to optimal threshold Dec 31 15:40:43.278693090 (624h11m43.149776145s) [noise=true , address=64.49.13.169:4706 , piece_id=W2I7DOJCVBAL7G37ZJMIZKY25T57OL5XSYWBB24PXY527LFFKTTA, seg_pos={0 0}, upload=546972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:39176->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:39176->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39176->64.49.13.169:4706: use of closed network connection Dec 31 15:40:43.278701535 (624h11m43.14978457s) [noise=true , address=64.49.13.169:4706 , piece_id=W2I7DOJCVBAL7G37ZJMIZKY25T57OL5XSYWBB24PXY527LFFKTTA, seg_pos={0 0}, upload=546972]: Long tail cancellation due to optimal threshold Dec 31 15:40:43.278813819 (624h11m43.149896864s) [seg_pos={0 0}, upload=546972]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:40:43.278915964 (624h11m43.149999009s) [upload=546972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:40:43.278921965 (624h11m43.15000501s) [upload=546972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:40:43.578211354 (624h11m43.449294399s) [upload=546972]: Done uploading segments... Dec 31 15:40:43.578242012 (624h11m43.449325047s) [upload=546972]: Upload finished. err: <nil> Duration: 1m0.004999693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [......X..XX...........XXX..X.XX..XX..XXXXX...X.X.X.......X.X] duration: 5.001444808s trace: https://jaeger.dev.storj.io/trace/6f1f4d3443d4ceb3 FAIL (2025-12-31 15:41:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00146684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.X.........X..........................] duration: 26.476912377s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:40:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 26.476942784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X..XX.] duration: 6.199590337s trace: https://jaeger.dev.storj.io/trace/1e9192a7c380565f FAIL (2025-12-31 15:40:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA) context deadline exceeded ======LOGS====== Dec 31 15:40:02.183200486 (624h11m2.054283531s) [upload=546961]: Starting upload Dec 31 15:40:02.183289697 (624h11m2.054372742s) [upload=546961]: Done starting upload Dec 31 15:40:02.183557467 (624h11m2.054640512s) [upload=546961]: Uploading segments... Dec 31 15:40:02.183626839 (624h11m2.054709924s) [upload=546961]: Got next segment. Inline: false Dec 31 15:40:02.183683035 (624h11m2.05476608s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:40:02.183691231 (624h11m2.054774276s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:40:02.433064484 (624h11m2.304147539s) [seg_pos={0 0}, upload=546961]: Begin upload segment... Dec 31 15:40:02.433122744 (624h11m2.304205799s) [seg_pos={0 0}, upload=546961]: Done begin upload segment. Dec 31 15:40:02.433159664 (624h11m2.304242719s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.434864581 (624h11m2.305947656s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.435672529 (624h11m2.306755574s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436304893 (624h11m2.307387938s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436829121 (624h11m2.307912166s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.438953957 (624h11m2.310037002s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.440128664 (624h11m2.311211709s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.445437173 (624h11m2.316520208s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.447396233 (624h11m2.318479288s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.448528338 (624h11m2.319611393s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.449941790 (624h11m2.321024855s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.452078879 (624h11m2.323161924s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.455900075 (624h11m2.32698315s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458138027 (624h11m2.329221072s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458601058 (624h11m2.329684103s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.460663175 (624h11m2.33174622s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:04.142907034 (624h11m4.013990079s) [upload=546961]: Next returned nil segment Dec 31 15:40:04.142928124 (624h11m4.014011169s) [upload=546961]: Waiting for error group managing segments... Dec 31 15:40:04.487357616 (624h11m4.358440661s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.505736883 (624h11m4.376819928s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.509642600 (624h11m4.380725645s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.511026896 (624h11m4.382109941s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516140091 (624h11m4.387223146s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516207069 (624h11m4.387290124s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.517863183 (624h11m4.388946228s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.554784937 (624h11m4.425867992s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.571180697 (624h11m4.442263732s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.593032015 (624h11m4.46411506s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.803506067 (624h11m4.674589112s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.818445714 (624h11m4.689528759s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856531133 (624h11m4.727614178s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856558596 (624h11m4.727641651s) [seg_pos={0 0}, upload=546961]: Segment reached optimal threshold of 13 pieces. Dec 31 15:40:04.856807299 (624h11m4.727890344s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.856819383 (624h11m4.727902418s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857304316 (624h11m4.728387372s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857313794 (624h11m4.728396839s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857472977 (624h11m4.728556032s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857481664 (624h11m4.728564709s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857504938 (624h11m4.728587983s) [seg_pos={0 0}, upload=546961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:40:04.857638522 (624h11m4.728721577s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:40:04.857646417 (624h11m4.728729462s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:40:05.159347058 (624h11m5.030430103s) [upload=546961]: Done uploading segments... Dec 31 15:40:05.159476614 (624h11m5.030559659s) [upload=546961]: Upload finished. err: <nil> Duration: 6.199606708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X..XX] duration: 30.003868795s trace: https://jaeger.dev.storj.io/trace/1e9192a7c380565f FAIL (2025-12-31 15:40:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA) context deadline exceeded ======LOGS====== Dec 31 15:40:02.183200486 (624h11m2.054283531s) [upload=546961]: Starting upload Dec 31 15:40:02.183289697 (624h11m2.054372742s) [upload=546961]: Done starting upload Dec 31 15:40:02.183557467 (624h11m2.054640512s) [upload=546961]: Uploading segments... Dec 31 15:40:02.183626839 (624h11m2.054709924s) [upload=546961]: Got next segment. Inline: false Dec 31 15:40:02.183683035 (624h11m2.05476608s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:40:02.183691231 (624h11m2.054774276s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:40:02.433064484 (624h11m2.304147539s) [seg_pos={0 0}, upload=546961]: Begin upload segment... Dec 31 15:40:02.433122744 (624h11m2.304205799s) [seg_pos={0 0}, upload=546961]: Done begin upload segment. Dec 31 15:40:02.433159664 (624h11m2.304242719s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.434864581 (624h11m2.305947656s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.435672529 (624h11m2.306755574s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436304893 (624h11m2.307387938s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.436829121 (624h11m2.307912166s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.438953957 (624h11m2.310037002s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.440128664 (624h11m2.311211709s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.445437173 (624h11m2.316520208s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.447396233 (624h11m2.318479288s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.448528338 (624h11m2.319611393s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.449941790 (624h11m2.321024855s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.452078879 (624h11m2.323161924s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.455900075 (624h11m2.32698315s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458138027 (624h11m2.329221072s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.458601058 (624h11m2.329684103s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:02.460663175 (624h11m2.33174622s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Uploading piece... Dec 31 15:40:04.142907034 (624h11m4.013990079s) [upload=546961]: Next returned nil segment Dec 31 15:40:04.142928124 (624h11m4.014011169s) [upload=546961]: Waiting for error group managing segments... Dec 31 15:40:04.487357616 (624h11m4.358440661s) [noise=true , address=145.79.227.27:4691 , piece_id=AVFFBYGQT2QN2U3ZQKIU7V7ABKVU4TCM2GZC456HH7R2CDKGZUOA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.505736883 (624h11m4.376819928s) [noise=true , address=145.79.227.37:4565 , piece_id=7TQ4CSZAVHTIONLCARS6BD3YGBF7FCT7MWYCM6QU7QDB2O4CHOMA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.509642600 (624h11m4.380725645s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZVCOFL45QUNHZFIKZBNM3YW2MQUDQ7PPYOTNVV3VZWBSTM372IQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.511026896 (624h11m4.382109941s) [noise=true , address=145.79.227.47:4266 , piece_id=Q32ZFKM7Q2SQVQ6MVER3RVNDYORG6KHDCHUIUCV7ISRKFZHWTFZQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516140091 (624h11m4.387223146s) [noise=true , address=145.79.227.23:4546 , piece_id=PKFZSB4XANJLIZTDH5V6JTZGZFXILDUHR55OKDDCFBNIPKDM3GGQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.516207069 (624h11m4.387290124s) [noise=true , address=145.79.227.42:4123 , piece_id=E7RZCK6KNL5LJXHXTTTDBEUKBAMSFVU3PKKBSMXO5CUPNKUK7KIQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.517863183 (624h11m4.388946228s) [noise=true , address=145.79.227.50:4375 , piece_id=IF6F2JRGDMNIB6WBVXR25RFHTFIHS4E6TXPOODXRJGP3LI4OM72Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.554784937 (624h11m4.425867992s) [noise=true , address=145.79.227.2:4114 , piece_id=CFVLY3NVA7FCZ2AFBUBKCM4CM5TO5BT3PNGH2KWDTWIY5N6JJB2Q, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.571180697 (624h11m4.442263732s) [noise=true , address=145.79.227.39:4013 , piece_id=TQPZ5NPPEL5RSS4KL6SGUDMYBZQGH36OGN7UVKAHOFFV73OAIFRQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.593032015 (624h11m4.46411506s) [noise=true , address=145.79.227.24:4582 , piece_id=2ZBHTJV2GMMO6EYQZLFNJM3USF3KCRWKWKVPQZY76FOGU7JQLCWA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.803506067 (624h11m4.674589112s) [noise=true , address=77.90.31.21:4880 , piece_id=NFDKL4ENLY3KYBXPZSLRHBWPZ7JFOZI3LQBJLT2LPJUD5HBK7QSA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.818445714 (624h11m4.689528759s) [noise=true , address=45.91.116.9:4262 , piece_id=G4N52AEZG6245HSSPFBKG53OHFIHHVARZC5ER2M3OEMT7FWVW2XQ, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856531133 (624h11m4.727614178s) [noise=true , address=45.91.116.8:4047 , piece_id=UN4F6JFYPKGPNTZL63KNVFYGMVSXYKQCCEWR7HFSXGEZHRYA3MHA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: <nil> Dec 31 15:40:04.856558596 (624h11m4.727641651s) [seg_pos={0 0}, upload=546961]: Segment reached optimal threshold of 13 pieces. Dec 31 15:40:04.856807299 (624h11m4.727890344s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.856819383 (624h11m4.727902418s) [noise=true , address=45.91.116.11:4119 , piece_id=C57GDBE674XWNGCIDXPGYBVGW6IAJXPVPCULTNNB4UOKBLYNCYQA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857304316 (624h11m4.728387372s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857313794 (624h11m4.728396839s) [noise=true , address=145.79.227.7:4294 , piece_id=ATHTQN755XA52DRTITZUAPB5VVXZLSWLWE3N4VLEFHH24EVFX52A, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857472977 (624h11m4.728556032s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 15:40:04.857481664 (624h11m4.728564709s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4DCT2LR3OOOT4VSYQTUI2OQBIUE2PGWDXK66PFNND64DHWGTRTA, seg_pos={0 0}, upload=546961]: Long tail cancellation due to optimal threshold Dec 31 15:40:04.857504938 (624h11m4.728587983s) [seg_pos={0 0}, upload=546961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:40:04.857638522 (624h11m4.728721577s) [upload=546961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:40:04.857646417 (624h11m4.728729462s) [upload=546961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:40:05.159347058 (624h11m5.030430103s) [upload=546961]: Done uploading segments... Dec 31 15:40:05.159476614 (624h11m5.030559659s) [upload=546961]: Upload finished. err: <nil> Duration: 30.003888943s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.......X..XX...........XXX..X.XX..XX..XXXXX...X.X.X.......X.] duration: 4.133711166s trace: https://jaeger.dev.storj.io/trace/48517d9132fa58ab FAIL (2025-12-31 15:40:21 UTC) orbiter warning: context deadline exceeded Duration: 4.13373968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.X.........X.........................] duration: 31.145809508s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:40:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 31.145837531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X..X] duration: 30.003234182s trace: https://jaeger.dev.storj.io/trace/4a50415d91685cd0 FAIL (2025-12-31 15:40:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 7WPSOG64RDU7Q5MWNJNAIHTX6RM23MYVSTYZLTEMVOSOHKYXG4OA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 6BQIA47HRYSDTGWFHZVYTPFOOL2FOI43YISMAWWIORTMNHMLKSNQ) context deadline exceeded ======LOGS====== Dec 31 15:39:31.924377907 (624h10m31.795460942s) [upload=546954]: Starting upload Dec 31 15:39:31.924441388 (624h10m31.795524423s) [upload=546954]: Done starting upload Dec 31 15:39:31.924661517 (624h10m31.795744552s) [upload=546954]: Uploading segments... Dec 31 15:39:31.924757600 (624h10m31.795840665s) [upload=546954]: Got next segment. Inline: false Dec 31 15:39:31.924805000 (624h10m31.795888035s) [upload=546954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:39:31.924810871 (624h10m31.795893916s) [upload=546954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:39:32.170730086 (624h10m32.041813121s) [seg_pos={0 0}, upload=546954]: Begin upload segment... Dec 31 15:39:32.170780171 (624h10m32.041863206s) [seg_pos={0 0}, upload=546954]: Done begin upload segment. Dec 31 15:39:32.170812432 (624h10m32.041895467s) [noise=true , address=45.91.116.10:4081 , piece_id=2TGWRZGQJMG7VIVY4WZRVHMI5HS2DXOP4IZGPSE4T7FVMAMLBMTQ, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.170835687 (624h10m32.041918722s) [noise=true , address=145.79.227.15:4874 , piece_id=ER7UQZVAGQGY2Y4I7GRAGESHFM546AJERC2TK2RQG4X5XJMZ3WFA, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.170877576 (624h10m32.041960612s) [noise=true , address=145.79.227.4:4185 , piece_id=ZBCNZ54R5JG5D2ZDJ6W7DPMEEWZXVJBKD6XVR2QAFWXJ3B6FJOHQ, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.171191524 (624h10m32.042274559s) [noise=true , address=145.79.227.31:4798 , piece_id=ASVGBTDIYLKSY6VCX7VYTUOFCO7TJVJWEDUAGQJDQTKY3VGVYYZA, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.171446790 (624h10m32.042529825s) [noise=true , address=145.79.227.41:4086 , piece_id=ZWGZGELM5Q3QDL6QOTSY4HWRQCN5MCILGWYGCP3TS5HNM7L4I4ZQ, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.171564243 (624h10m32.042647278s) [noise=true , address=45.91.116.5:4298 , piece_id=6BQIA47HRYSDTGWFHZVYTPFOOL2FOI43YISMAWWIORTMNHMLKSNQ, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.171695684 (624h10m32.042778719s) [noise=true , address=45.91.116.12:4227 , piece_id=GCE6O53IHVXBZSGQFR2R4QA2Q4IYE2ZRRCRFVWWCJTPIZWIYPBUQ, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.171944347 (624h10m32.043027382s) [noise=true , address=145.79.227.2:4115 , piece_id=DAMP5IBQXCBA2WXSDCRAHLQDUMWVDQ6NLSYUVVEY4HYSIUMZJVGQ, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.172273064 (624h10m32.043356099s) [noise=true , address=77.90.31.21:4878 , piece_id=XHBXZSN4PLEJC6KYEZWAPJ2FWQFXOMJVVPR2LGQX73N7GW7A4O2A, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.172544831 (624h10m32.043627866s) [noise=true , address=145.79.227.23:4547 , piece_id=7WYFHZ3TZU5NMLMG7KZG25ZVZ3GJFL5L6OV63FHO6BTKNFA4VAHQ, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.172797632 (624h10m32.043880667s) [noise=true , address=145.79.227.46:4603 , piece_id=64FFFH52BSZYYRTY3PPO6GHB4GBXNY5ZCAC6VE44IW3MERI6RK7Q, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.173178668 (624h10m32.044261703s) [noise=true , address=145.79.227.14:4839 , piece_id=ZYMEG77KLFE2U5CGKECOMU3ZIVL6GIOJOLM5BRTUV6C6ZBKEEQ4Q, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.173471145 (624h10m32.04455418s) [noise=true , address=77.90.31.15:4700 , piece_id=UXDOEVOX3X5FXL252TZO2CXXI52JNX62WNRGIL4LWNIFEDMZ4PRA, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.173742061 (624h10m32.044825096s) [noise=true , address=45.91.116.9:4263 , piece_id=7WPSOG64RDU7Q5MWNJNAIHTX6RM23MYVSTYZLTEMVOSOHKYXG4OA, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.174066508 (624h10m32.045149543s) [noise=true , address=145.79.227.19:4079 , piece_id=JBZTPPEEA2G7LTLWVOG753UD3G7EZEFJFY3II7MKVZ6EXBM4QG4Q, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:32.174423469 (624h10m32.045506514s) [noise=true , address=45.91.116.13:4156 , piece_id=B3MSAZOOQQGDZ4QKYPWNX6CWY22KGN5NUQQ5T5MW5CSE2KE4C5QQ, seg_pos={0 0}, upload=546954]: Uploading piece... Dec 31 15:39:33.878652680 (624h10m33.749735725s) [upload=546954]: Next returned nil segment Dec 31 15:39:33.878665114 (624h10m33.749748149s) [upload=546954]: Waiting for error group managing segments... Dec 31 15:39:34.159122644 (624h10m34.030205679s) [noise=true , address=145.79.227.2:4115 , piece_id=DAMP5IBQXCBA2WXSDCRAHLQDUMWVDQ6NLSYUVVEY4HYSIUMZJVGQ, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.160054568 (624h10m34.031137613s) [noise=true , address=145.79.227.19:4079 , piece_id=JBZTPPEEA2G7LTLWVOG753UD3G7EZEFJFY3II7MKVZ6EXBM4QG4Q, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.172813433 (624h10m34.043896468s) [noise=true , address=145.79.227.4:4185 , piece_id=ZBCNZ54R5JG5D2ZDJ6W7DPMEEWZXVJBKD6XVR2QAFWXJ3B6FJOHQ, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.177030924 (624h10m34.048113959s) [noise=true , address=145.79.227.31:4798 , piece_id=ASVGBTDIYLKSY6VCX7VYTUOFCO7TJVJWEDUAGQJDQTKY3VGVYYZA, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.177596851 (624h10m34.048679896s) [noise=true , address=145.79.227.15:4874 , piece_id=ER7UQZVAGQGY2Y4I7GRAGESHFM546AJERC2TK2RQG4X5XJMZ3WFA, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.181115391 (624h10m34.052198436s) [noise=true , address=145.79.227.46:4603 , piece_id=64FFFH52BSZYYRTY3PPO6GHB4GBXNY5ZCAC6VE44IW3MERI6RK7Q, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.226100719 (624h10m34.097183764s) [noise=true , address=145.79.227.14:4839 , piece_id=ZYMEG77KLFE2U5CGKECOMU3ZIVL6GIOJOLM5BRTUV6C6ZBKEEQ4Q, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.242943451 (624h10m34.114026496s) [noise=true , address=145.79.227.41:4086 , piece_id=ZWGZGELM5Q3QDL6QOTSY4HWRQCN5MCILGWYGCP3TS5HNM7L4I4ZQ, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.249930896 (624h10m34.121013931s) [noise=true , address=145.79.227.23:4547 , piece_id=7WYFHZ3TZU5NMLMG7KZG25ZVZ3GJFL5L6OV63FHO6BTKNFA4VAHQ, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.478617988 (624h10m34.349701023s) [noise=true , address=45.91.116.13:4156 , piece_id=B3MSAZOOQQGDZ4QKYPWNX6CWY22KGN5NUQQ5T5MW5CSE2KE4C5QQ, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.580689647 (624h10m34.451772692s) [noise=true , address=45.91.116.5:4298 , piece_id=6BQIA47HRYSDTGWFHZVYTPFOOL2FOI43YISMAWWIORTMNHMLKSNQ, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:34.591907788 (624h10m34.462990823s) [noise=true , address=45.91.116.9:4263 , piece_id=7WPSOG64RDU7Q5MWNJNAIHTX6RM23MYVSTYZLTEMVOSOHKYXG4OA, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:35.070879627 (624h10m34.941962662s) [noise=true , address=77.90.31.21:4878 , piece_id=XHBXZSN4PLEJC6KYEZWAPJ2FWQFXOMJVVPR2LGQX73N7GW7A4O2A, seg_pos={0 0}, upload=546954]: Done uploading piece. err: <nil> Dec 31 15:39:35.070898493 (624h10m34.941981528s) [seg_pos={0 0}, upload=546954]: Segment reached optimal threshold of 13 pieces. Dec 31 15:39:35.071066473 (624h10m34.942149518s) [noise=true , address=45.91.116.12:4227 , piece_id=GCE6O53IHVXBZSGQFR2R4QA2Q4IYE2ZRRCRFVWWCJTPIZWIYPBUQ, seg_pos={0 0}, upload=546954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 15:39:35.071073757 (624h10m34.942156792s) [noise=true , address=45.91.116.12:4227 , piece_id=GCE6O53IHVXBZSGQFR2R4QA2Q4IYE2ZRRCRFVWWCJTPIZWIYPBUQ, seg_pos={0 0}, upload=546954]: Long tail cancellation due to optimal threshold Dec 31 15:39:35.071272155 (624h10m34.94235519s) [noise=true , address=77.90.31.15:4700 , piece_id=UXDOEVOX3X5FXL252TZO2CXXI52JNX62WNRGIL4LWNIFEDMZ4PRA, seg_pos={0 0}, upload=546954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 31 15:39:35.071277795 (624h10m34.94236083s) [noise=true , address=77.90.31.15:4700 , piece_id=UXDOEVOX3X5FXL252TZO2CXXI52JNX62WNRGIL4LWNIFEDMZ4PRA, seg_pos={0 0}, upload=546954]: Long tail cancellation due to optimal threshold Dec 31 15:39:35.071422561 (624h10m34.942505606s) [noise=true , address=45.91.116.10:4081 , piece_id=2TGWRZGQJMG7VIVY4WZRVHMI5HS2DXOP4IZGPSE4T7FVMAMLBMTQ, seg_pos={0 0}, upload=546954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 15:39:35.071431437 (624h10m34.942514472s) [noise=true , address=45.91.116.10:4081 , piece_id=2TGWRZGQJMG7VIVY4WZRVHMI5HS2DXOP4IZGPSE4T7FVMAMLBMTQ, seg_pos={0 0}, upload=546954]: Long tail cancellation due to optimal threshold Dec 31 15:39:35.071448791 (624h10m34.942531836s) [seg_pos={0 0}, upload=546954]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:39:35.071545154 (624h10m34.942628189s) [upload=546954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:39:35.071549733 (624h10m34.942632768s) [upload=546954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:39:35.365291678 (624h10m35.236374723s) [upload=546954]: Done uploading segments... Dec 31 15:39:35.365312218 (624h10m35.236395253s) [upload=546954]: Upload finished. err: <nil> Duration: 30.00326464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.X.........X........................] duration: 32.598059538s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:39:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 32.598080059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [........X..XX...........XXX..X.XX..XX..XXXXX...X.X.X.......X] duration: 5.001202434s trace: https://jaeger.dev.storj.io/trace/48517d9132fa58ab FAIL (2025-12-31 15:39:16 UTC) orbiter warning: context deadline exceeded Duration: 5.001227263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X..] duration: 6.071491117s trace: https://jaeger.dev.storj.io/trace/782d06891e0d0edb FAIL (2025-12-31 15:38:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA) context deadline exceeded ======LOGS====== Dec 31 15:37:36.307875141 (624h8m36.178958176s) [upload=546929]: Starting upload Dec 31 15:37:36.308071906 (624h8m36.179154971s) [upload=546929]: Done starting upload Dec 31 15:37:36.308282537 (624h8m36.179365582s) [upload=546929]: Uploading segments... Dec 31 15:37:36.308441389 (624h8m36.179524454s) [upload=546929]: Got next segment. Inline: false Dec 31 15:37:36.308494791 (624h8m36.179577826s) [upload=546929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:37:36.308501494 (624h8m36.179584529s) [upload=546929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:37:36.555242052 (624h8m36.426325087s) [seg_pos={0 0}, upload=546929]: Begin upload segment... Dec 31 15:37:36.555291436 (624h8m36.426374471s) [seg_pos={0 0}, upload=546929]: Done begin upload segment. Dec 31 15:37:36.555320902 (624h8m36.426403947s) [noise=true , address=77.90.31.15:4700 , piece_id=YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.555437094 (624h8m36.426520149s) [noise=true , address=45.91.116.7:4009 , piece_id=EEO6735UFJ3UXMZ6AXMNISGMPHN6I3K7WO32JH25WEPGJQPTCJGQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556388926 (624h8m36.427471971s) [noise=true , address=145.79.227.19:4077 , piece_id=J2TNGWJXD3FIOZ2KNROQW2JVSUWMZLEJRNRBGXITL7XCROYZBZRA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556513463 (624h8m36.427596498s) [noise=true , address=145.79.227.52:4445 , piece_id=ZXLUK4IQUOCQBXAD36OOVBICAUOXPYKOFFOS2BCMSBIIE4N3FIPA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556982617 (624h8m36.428065652s) [noise=true , address=145.79.227.13:4802 , piece_id=LMSECJUX7ZPNGKQDHLE5BSLUPWEJYEQGGCIHPCL5WWYFSGNKSAXA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.557446660 (624h8m36.428529705s) [noise=true , address=45.91.116.9:4263 , piece_id=E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.557941202 (624h8m36.429024247s) [noise=true , address=145.79.227.15:4873 , piece_id=EDPLYAQ2UJLLQZXM6VF3VVWYZNYHIIAAPT6UI2BE3F2M6QUW7EIQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558170358 (624h8m36.429253443s) [noise=true , address=145.79.227.23:4546 , piece_id=V3I7WKBQ2P2LIQ3EOBGIER7PDPIJDCJS6I6D7PCPOH6KYR3W46WA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558559800 (624h8m36.429642845s) [noise=true , address=145.79.227.22:4509 , piece_id=N5D5FCJUSNL5DJ4VRM4EWWQDRC423ZAFRQRHTQOPFMTTYZVQIWNA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558740714 (624h8m36.429823759s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558993346 (624h8m36.430076381s) [noise=true , address=145.79.227.46:4601 , piece_id=CXDCMU4UZEKGXZO5Y5B3B7BDOWR4IRPIVLACQYZUTUNHYO4UQMQQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559121459 (624h8m36.430204494s) [noise=true , address=145.79.227.14:4837 , piece_id=CUDARFRNM62WGUQDTS4NNI3WZWEC6FEZ7FQSQ7564EPUJTQFT32Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559417664 (624h8m36.430500699s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559551037 (624h8m36.430634072s) [noise=true , address=145.79.227.31:4798 , piece_id=3YTMLVSUCQRUFRKA5APOKLJTOJWQ56H6YC2HVEDV3IZHSP2HI24A, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559933265 (624h8m36.43101631s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.560478533 (624h8m36.431561578s) [noise=true , address=145.79.227.4:4187 , piece_id=LQROJKXN4NVCKGT4VVJZMGNOJEM7TIGVLLKQL42Y5WCBK7V2AP6Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:38.248307376 (624h8m38.119390411s) [upload=546929]: Next returned nil segment Dec 31 15:37:38.248329579 (624h8m38.119412634s) [upload=546929]: Waiting for error group managing segments... Dec 31 15:37:38.534776974 (624h8m38.405860019s) [noise=true , address=145.79.227.31:4798 , piece_id=3YTMLVSUCQRUFRKA5APOKLJTOJWQ56H6YC2HVEDV3IZHSP2HI24A, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.535373079 (624h8m38.406456114s) [noise=true , address=145.79.227.14:4837 , piece_id=CUDARFRNM62WGUQDTS4NNI3WZWEC6FEZ7FQSQ7564EPUJTQFT32Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.538251661 (624h8m38.409334706s) [noise=true , address=145.79.227.15:4873 , piece_id=EDPLYAQ2UJLLQZXM6VF3VVWYZNYHIIAAPT6UI2BE3F2M6QUW7EIQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.541587181 (624h8m38.412670216s) [noise=true , address=145.79.227.13:4802 , piece_id=LMSECJUX7ZPNGKQDHLE5BSLUPWEJYEQGGCIHPCL5WWYFSGNKSAXA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.545723909 (624h8m38.416806944s) [noise=true , address=145.79.227.22:4509 , piece_id=N5D5FCJUSNL5DJ4VRM4EWWQDRC423ZAFRQRHTQOPFMTTYZVQIWNA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.554291352 (624h8m38.425374387s) [noise=true , address=145.79.227.19:4077 , piece_id=J2TNGWJXD3FIOZ2KNROQW2JVSUWMZLEJRNRBGXITL7XCROYZBZRA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.558667115 (624h8m38.42975016s) [noise=true , address=145.79.227.52:4445 , piece_id=ZXLUK4IQUOCQBXAD36OOVBICAUOXPYKOFFOS2BCMSBIIE4N3FIPA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.565262643 (624h8m38.436345688s) [noise=true , address=145.79.227.4:4187 , piece_id=LQROJKXN4NVCKGT4VVJZMGNOJEM7TIGVLLKQL42Y5WCBK7V2AP6Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.566431990 (624h8m38.437515035s) [noise=true , address=145.79.227.23:4546 , piece_id=V3I7WKBQ2P2LIQ3EOBGIER7PDPIJDCJS6I6D7PCPOH6KYR3W46WA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.615150386 (624h8m38.486233421s) [noise=true , address=145.79.227.46:4601 , piece_id=CXDCMU4UZEKGXZO5Y5B3B7BDOWR4IRPIVLACQYZUTUNHYO4UQMQQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.731035007 (624h8m38.602118043s) [noise=true , address=77.90.31.15:4700 , piece_id=YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.852793472 (624h8m38.723876507s) [noise=true , address=45.91.116.9:4263 , piece_id=E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.899290520 (624h8m38.770373565s) [noise=true , address=45.91.116.7:4009 , piece_id=EEO6735UFJ3UXMZ6AXMNISGMPHN6I3K7WO32JH25WEPGJQPTCJGQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.899316057 (624h8m38.770399093s) [seg_pos={0 0}, upload=546929]: Segment reached optimal threshold of 13 pieces. Dec 31 15:37:38.899664932 (624h8m38.770747977s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 15:37:38.899680642 (624h8m38.770763687s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899850596 (624h8m38.770933641s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 15:37:38.899858982 (624h8m38.770942017s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899876174 (624h8m38.770959209s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection Dec 31 15:37:38.899886254 (624h8m38.770969299s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899904187 (624h8m38.770987223s) [seg_pos={0 0}, upload=546929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:37:38.899984460 (624h8m38.771067495s) [upload=546929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:37:38.899991134 (624h8m38.771074169s) [upload=546929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:37:39.193357724 (624h8m39.064440769s) [upload=546929]: Done uploading segments... Dec 31 15:37:39.193380377 (624h8m39.064463412s) [upload=546929]: Upload finished. err: <nil> Duration: 6.071518359s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.........X..XX...........XXX..X.XX..XX..XXXXX...X.X.X.......] duration: 3.877275978s trace: https://jaeger.dev.storj.io/trace/1901bc26af4982d6 FAIL (2025-12-31 15:38:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.877313498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.X.........X.......................] duration: 30.751014067s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:38:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 30.751031408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X.] duration: 5.759753062s trace: https://jaeger.dev.storj.io/trace/782d06891e0d0edb FAIL (2025-12-31 15:38:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA) context deadline exceeded ======LOGS====== Dec 31 15:37:36.307875141 (624h8m36.178958176s) [upload=546929]: Starting upload Dec 31 15:37:36.308071906 (624h8m36.179154971s) [upload=546929]: Done starting upload Dec 31 15:37:36.308282537 (624h8m36.179365582s) [upload=546929]: Uploading segments... Dec 31 15:37:36.308441389 (624h8m36.179524454s) [upload=546929]: Got next segment. Inline: false Dec 31 15:37:36.308494791 (624h8m36.179577826s) [upload=546929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:37:36.308501494 (624h8m36.179584529s) [upload=546929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:37:36.555242052 (624h8m36.426325087s) [seg_pos={0 0}, upload=546929]: Begin upload segment... Dec 31 15:37:36.555291436 (624h8m36.426374471s) [seg_pos={0 0}, upload=546929]: Done begin upload segment. Dec 31 15:37:36.555320902 (624h8m36.426403947s) [noise=true , address=77.90.31.15:4700 , piece_id=YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.555437094 (624h8m36.426520149s) [noise=true , address=45.91.116.7:4009 , piece_id=EEO6735UFJ3UXMZ6AXMNISGMPHN6I3K7WO32JH25WEPGJQPTCJGQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556388926 (624h8m36.427471971s) [noise=true , address=145.79.227.19:4077 , piece_id=J2TNGWJXD3FIOZ2KNROQW2JVSUWMZLEJRNRBGXITL7XCROYZBZRA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556513463 (624h8m36.427596498s) [noise=true , address=145.79.227.52:4445 , piece_id=ZXLUK4IQUOCQBXAD36OOVBICAUOXPYKOFFOS2BCMSBIIE4N3FIPA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556982617 (624h8m36.428065652s) [noise=true , address=145.79.227.13:4802 , piece_id=LMSECJUX7ZPNGKQDHLE5BSLUPWEJYEQGGCIHPCL5WWYFSGNKSAXA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.557446660 (624h8m36.428529705s) [noise=true , address=45.91.116.9:4263 , piece_id=E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.557941202 (624h8m36.429024247s) [noise=true , address=145.79.227.15:4873 , piece_id=EDPLYAQ2UJLLQZXM6VF3VVWYZNYHIIAAPT6UI2BE3F2M6QUW7EIQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558170358 (624h8m36.429253443s) [noise=true , address=145.79.227.23:4546 , piece_id=V3I7WKBQ2P2LIQ3EOBGIER7PDPIJDCJS6I6D7PCPOH6KYR3W46WA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558559800 (624h8m36.429642845s) [noise=true , address=145.79.227.22:4509 , piece_id=N5D5FCJUSNL5DJ4VRM4EWWQDRC423ZAFRQRHTQOPFMTTYZVQIWNA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558740714 (624h8m36.429823759s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558993346 (624h8m36.430076381s) [noise=true , address=145.79.227.46:4601 , piece_id=CXDCMU4UZEKGXZO5Y5B3B7BDOWR4IRPIVLACQYZUTUNHYO4UQMQQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559121459 (624h8m36.430204494s) [noise=true , address=145.79.227.14:4837 , piece_id=CUDARFRNM62WGUQDTS4NNI3WZWEC6FEZ7FQSQ7564EPUJTQFT32Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559417664 (624h8m36.430500699s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559551037 (624h8m36.430634072s) [noise=true , address=145.79.227.31:4798 , piece_id=3YTMLVSUCQRUFRKA5APOKLJTOJWQ56H6YC2HVEDV3IZHSP2HI24A, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559933265 (624h8m36.43101631s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.560478533 (624h8m36.431561578s) [noise=true , address=145.79.227.4:4187 , piece_id=LQROJKXN4NVCKGT4VVJZMGNOJEM7TIGVLLKQL42Y5WCBK7V2AP6Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:38.248307376 (624h8m38.119390411s) [upload=546929]: Next returned nil segment Dec 31 15:37:38.248329579 (624h8m38.119412634s) [upload=546929]: Waiting for error group managing segments... Dec 31 15:37:38.534776974 (624h8m38.405860019s) [noise=true , address=145.79.227.31:4798 , piece_id=3YTMLVSUCQRUFRKA5APOKLJTOJWQ56H6YC2HVEDV3IZHSP2HI24A, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.535373079 (624h8m38.406456114s) [noise=true , address=145.79.227.14:4837 , piece_id=CUDARFRNM62WGUQDTS4NNI3WZWEC6FEZ7FQSQ7564EPUJTQFT32Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.538251661 (624h8m38.409334706s) [noise=true , address=145.79.227.15:4873 , piece_id=EDPLYAQ2UJLLQZXM6VF3VVWYZNYHIIAAPT6UI2BE3F2M6QUW7EIQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.541587181 (624h8m38.412670216s) [noise=true , address=145.79.227.13:4802 , piece_id=LMSECJUX7ZPNGKQDHLE5BSLUPWEJYEQGGCIHPCL5WWYFSGNKSAXA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.545723909 (624h8m38.416806944s) [noise=true , address=145.79.227.22:4509 , piece_id=N5D5FCJUSNL5DJ4VRM4EWWQDRC423ZAFRQRHTQOPFMTTYZVQIWNA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.554291352 (624h8m38.425374387s) [noise=true , address=145.79.227.19:4077 , piece_id=J2TNGWJXD3FIOZ2KNROQW2JVSUWMZLEJRNRBGXITL7XCROYZBZRA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.558667115 (624h8m38.42975016s) [noise=true , address=145.79.227.52:4445 , piece_id=ZXLUK4IQUOCQBXAD36OOVBICAUOXPYKOFFOS2BCMSBIIE4N3FIPA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.565262643 (624h8m38.436345688s) [noise=true , address=145.79.227.4:4187 , piece_id=LQROJKXN4NVCKGT4VVJZMGNOJEM7TIGVLLKQL42Y5WCBK7V2AP6Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.566431990 (624h8m38.437515035s) [noise=true , address=145.79.227.23:4546 , piece_id=V3I7WKBQ2P2LIQ3EOBGIER7PDPIJDCJS6I6D7PCPOH6KYR3W46WA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.615150386 (624h8m38.486233421s) [noise=true , address=145.79.227.46:4601 , piece_id=CXDCMU4UZEKGXZO5Y5B3B7BDOWR4IRPIVLACQYZUTUNHYO4UQMQQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.731035007 (624h8m38.602118043s) [noise=true , address=77.90.31.15:4700 , piece_id=YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.852793472 (624h8m38.723876507s) [noise=true , address=45.91.116.9:4263 , piece_id=E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.899290520 (624h8m38.770373565s) [noise=true , address=45.91.116.7:4009 , piece_id=EEO6735UFJ3UXMZ6AXMNISGMPHN6I3K7WO32JH25WEPGJQPTCJGQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.899316057 (624h8m38.770399093s) [seg_pos={0 0}, upload=546929]: Segment reached optimal threshold of 13 pieces. Dec 31 15:37:38.899664932 (624h8m38.770747977s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 15:37:38.899680642 (624h8m38.770763687s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899850596 (624h8m38.770933641s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 15:37:38.899858982 (624h8m38.770942017s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899876174 (624h8m38.770959209s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection Dec 31 15:37:38.899886254 (624h8m38.770969299s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899904187 (624h8m38.770987223s) [seg_pos={0 0}, upload=546929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:37:38.899984460 (624h8m38.771067495s) [upload=546929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:37:38.899991134 (624h8m38.771074169s) [upload=546929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:37:39.193357724 (624h8m39.064440769s) [upload=546929]: Done uploading segments... Dec 31 15:37:39.193380377 (624h8m39.064463412s) [upload=546929]: Upload finished. err: <nil> Duration: 5.759779693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.X] duration: 30.003290663s trace: https://jaeger.dev.storj.io/trace/782d06891e0d0edb FAIL (2025-12-31 15:38:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA) context deadline exceeded ======LOGS====== Dec 31 15:37:36.307875141 (624h8m36.178958176s) [upload=546929]: Starting upload Dec 31 15:37:36.308071906 (624h8m36.179154971s) [upload=546929]: Done starting upload Dec 31 15:37:36.308282537 (624h8m36.179365582s) [upload=546929]: Uploading segments... Dec 31 15:37:36.308441389 (624h8m36.179524454s) [upload=546929]: Got next segment. Inline: false Dec 31 15:37:36.308494791 (624h8m36.179577826s) [upload=546929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:37:36.308501494 (624h8m36.179584529s) [upload=546929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:37:36.555242052 (624h8m36.426325087s) [seg_pos={0 0}, upload=546929]: Begin upload segment... Dec 31 15:37:36.555291436 (624h8m36.426374471s) [seg_pos={0 0}, upload=546929]: Done begin upload segment. Dec 31 15:37:36.555320902 (624h8m36.426403947s) [noise=true , address=77.90.31.15:4700 , piece_id=YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.555437094 (624h8m36.426520149s) [noise=true , address=45.91.116.7:4009 , piece_id=EEO6735UFJ3UXMZ6AXMNISGMPHN6I3K7WO32JH25WEPGJQPTCJGQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556388926 (624h8m36.427471971s) [noise=true , address=145.79.227.19:4077 , piece_id=J2TNGWJXD3FIOZ2KNROQW2JVSUWMZLEJRNRBGXITL7XCROYZBZRA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556513463 (624h8m36.427596498s) [noise=true , address=145.79.227.52:4445 , piece_id=ZXLUK4IQUOCQBXAD36OOVBICAUOXPYKOFFOS2BCMSBIIE4N3FIPA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.556982617 (624h8m36.428065652s) [noise=true , address=145.79.227.13:4802 , piece_id=LMSECJUX7ZPNGKQDHLE5BSLUPWEJYEQGGCIHPCL5WWYFSGNKSAXA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.557446660 (624h8m36.428529705s) [noise=true , address=45.91.116.9:4263 , piece_id=E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.557941202 (624h8m36.429024247s) [noise=true , address=145.79.227.15:4873 , piece_id=EDPLYAQ2UJLLQZXM6VF3VVWYZNYHIIAAPT6UI2BE3F2M6QUW7EIQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558170358 (624h8m36.429253443s) [noise=true , address=145.79.227.23:4546 , piece_id=V3I7WKBQ2P2LIQ3EOBGIER7PDPIJDCJS6I6D7PCPOH6KYR3W46WA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558559800 (624h8m36.429642845s) [noise=true , address=145.79.227.22:4509 , piece_id=N5D5FCJUSNL5DJ4VRM4EWWQDRC423ZAFRQRHTQOPFMTTYZVQIWNA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558740714 (624h8m36.429823759s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.558993346 (624h8m36.430076381s) [noise=true , address=145.79.227.46:4601 , piece_id=CXDCMU4UZEKGXZO5Y5B3B7BDOWR4IRPIVLACQYZUTUNHYO4UQMQQ, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559121459 (624h8m36.430204494s) [noise=true , address=145.79.227.14:4837 , piece_id=CUDARFRNM62WGUQDTS4NNI3WZWEC6FEZ7FQSQ7564EPUJTQFT32Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559417664 (624h8m36.430500699s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559551037 (624h8m36.430634072s) [noise=true , address=145.79.227.31:4798 , piece_id=3YTMLVSUCQRUFRKA5APOKLJTOJWQ56H6YC2HVEDV3IZHSP2HI24A, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.559933265 (624h8m36.43101631s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:36.560478533 (624h8m36.431561578s) [noise=true , address=145.79.227.4:4187 , piece_id=LQROJKXN4NVCKGT4VVJZMGNOJEM7TIGVLLKQL42Y5WCBK7V2AP6Q, seg_pos={0 0}, upload=546929]: Uploading piece... Dec 31 15:37:38.248307376 (624h8m38.119390411s) [upload=546929]: Next returned nil segment Dec 31 15:37:38.248329579 (624h8m38.119412634s) [upload=546929]: Waiting for error group managing segments... Dec 31 15:37:38.534776974 (624h8m38.405860019s) [noise=true , address=145.79.227.31:4798 , piece_id=3YTMLVSUCQRUFRKA5APOKLJTOJWQ56H6YC2HVEDV3IZHSP2HI24A, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.535373079 (624h8m38.406456114s) [noise=true , address=145.79.227.14:4837 , piece_id=CUDARFRNM62WGUQDTS4NNI3WZWEC6FEZ7FQSQ7564EPUJTQFT32Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.538251661 (624h8m38.409334706s) [noise=true , address=145.79.227.15:4873 , piece_id=EDPLYAQ2UJLLQZXM6VF3VVWYZNYHIIAAPT6UI2BE3F2M6QUW7EIQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.541587181 (624h8m38.412670216s) [noise=true , address=145.79.227.13:4802 , piece_id=LMSECJUX7ZPNGKQDHLE5BSLUPWEJYEQGGCIHPCL5WWYFSGNKSAXA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.545723909 (624h8m38.416806944s) [noise=true , address=145.79.227.22:4509 , piece_id=N5D5FCJUSNL5DJ4VRM4EWWQDRC423ZAFRQRHTQOPFMTTYZVQIWNA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.554291352 (624h8m38.425374387s) [noise=true , address=145.79.227.19:4077 , piece_id=J2TNGWJXD3FIOZ2KNROQW2JVSUWMZLEJRNRBGXITL7XCROYZBZRA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.558667115 (624h8m38.42975016s) [noise=true , address=145.79.227.52:4445 , piece_id=ZXLUK4IQUOCQBXAD36OOVBICAUOXPYKOFFOS2BCMSBIIE4N3FIPA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.565262643 (624h8m38.436345688s) [noise=true , address=145.79.227.4:4187 , piece_id=LQROJKXN4NVCKGT4VVJZMGNOJEM7TIGVLLKQL42Y5WCBK7V2AP6Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.566431990 (624h8m38.437515035s) [noise=true , address=145.79.227.23:4546 , piece_id=V3I7WKBQ2P2LIQ3EOBGIER7PDPIJDCJS6I6D7PCPOH6KYR3W46WA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.615150386 (624h8m38.486233421s) [noise=true , address=145.79.227.46:4601 , piece_id=CXDCMU4UZEKGXZO5Y5B3B7BDOWR4IRPIVLACQYZUTUNHYO4UQMQQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.731035007 (624h8m38.602118043s) [noise=true , address=77.90.31.15:4700 , piece_id=YAHHLFXDV3CQXEBNIEPHAOCUHVHJIXTJD2Z2IJKSQSXEOFF3RJRQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.852793472 (624h8m38.723876507s) [noise=true , address=45.91.116.9:4263 , piece_id=E6DEP2VC6TIW5XKMSN6SAPQN7GND5GOVQPBJS3J4RC5QKVLSG3KA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.899290520 (624h8m38.770373565s) [noise=true , address=45.91.116.7:4009 , piece_id=EEO6735UFJ3UXMZ6AXMNISGMPHN6I3K7WO32JH25WEPGJQPTCJGQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: <nil> Dec 31 15:37:38.899316057 (624h8m38.770399093s) [seg_pos={0 0}, upload=546929]: Segment reached optimal threshold of 13 pieces. Dec 31 15:37:38.899664932 (624h8m38.770747977s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 31 15:37:38.899680642 (624h8m38.770763687s) [noise=true , address=45.91.116.5:4297 , piece_id=QXRQV2GQY3CR6RU3QAMD3G57N72IUMUE7YUWYBQ5ZW32N6BBNLEQ, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899850596 (624h8m38.770933641s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 15:37:38.899858982 (624h8m38.770942017s) [noise=true , address=45.91.116.12:4228 , piece_id=SHRIL7PX3OOQUQJGKZYG6EZDMCYLRSI2JCJLR2FHOHM6JYLD2HKA, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899876174 (624h8m38.770959209s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57766->45.91.116.13:4155: use of closed network connection Dec 31 15:37:38.899886254 (624h8m38.770969299s) [noise=true , address=45.91.116.13:4155 , piece_id=VRZ2PSEKQNUYGF7QG7BOQUWDZF65SJO7UAPHDEZHORZHQKOLSW3Q, seg_pos={0 0}, upload=546929]: Long tail cancellation due to optimal threshold Dec 31 15:37:38.899904187 (624h8m38.770987223s) [seg_pos={0 0}, upload=546929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:37:38.899984460 (624h8m38.771067495s) [upload=546929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:37:38.899991134 (624h8m38.771074169s) [upload=546929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:37:39.193357724 (624h8m39.064440769s) [upload=546929]: Done uploading segments... Dec 31 15:37:39.193380377 (624h8m39.064463412s) [upload=546929]: Upload finished. err: <nil> Duration: 30.003304379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.X.........X......................] duration: 26.30107183s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:37:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 26.301100844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..........X..XX...........XXX..X.XX..XX..XXXXX...X.X.X......] duration: 3.709772487s trace: https://jaeger.dev.storj.io/trace/1901bc26af4982d6 FAIL (2025-12-31 15:37:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.709799618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X.] duration: 5.83400366s trace: https://jaeger.dev.storj.io/trace/2d9b9761b902a2f6 FAIL (2025-12-31 15:37:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: ZX73DOP2WJU3FLD6KBIZBZXK4LWML2NGURRA6QLR7P3BCPFAG3VA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: JON75QF3WXYTC6MF34PWDUUUPJ632D3S4TCJF7CNN7ZALIBURAOA) context deadline exceeded ======LOGS====== 624h7m48.112592241s) [upload=546917]: Uploading segments... Dec 31 15:36:48.241677396 (624h7m48.112760431s) [upload=546917]: Got next segment. Inline: false Dec 31 15:36:48.241749483 (624h7m48.112832519s) [upload=546917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:36:48.241757859 (624h7m48.112840924s) [upload=546917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:36:48.490069609 (624h7m48.361152655s) [seg_pos={0 0}, upload=546917]: Begin upload segment... Dec 31 15:36:48.490125506 (624h7m48.361208551s) [seg_pos={0 0}, upload=546917]: Done begin upload segment. Dec 31 15:36:48.490155103 (624h7m48.361238158s) [noise=true , address=145.79.227.20:4439 , piece_id=AR2RRIAIWIETLYUFBCX6KUFNZ72EYOUCVDV4FWL7OAITLATG3M4Q, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.490211570 (624h7m48.361294615s) [noise=true , address=77.90.31.21:4877 , piece_id=ZX73DOP2WJU3FLD6KBIZBZXK4LWML2NGURRA6QLR7P3BCPFAG3VA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.490968261 (624h7m48.362051306s) [noise=true , address=77.90.31.19:4842 , piece_id=JON75QF3WXYTC6MF34PWDUUUPJ632D3S4TCJF7CNN7ZALIBURAOA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.491043384 (624h7m48.362126439s) [noise=true , address=145.79.227.9:4365 , piece_id=MXQ3RZ7RMQIPF56JZ2L5ZRPUQKXUROCFU3I5MF43TO3RLNESAMIQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.491377240 (624h7m48.362460275s) [noise=true , address=145.79.227.51:4410 , piece_id=DKXRCAHY3BLW5UNYJLUTDKNMVSMYBRRQPUC4QJ4J64DNNVVHKVQA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.491757414 (624h7m48.362840469s) [noise=true , address=145.79.227.25:4619 , piece_id=S5WLABYCZ6VLTRZVEAK7K4RAWOM5EFSSHYAEHRRA6DI6FSRX3XKQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.492060009 (624h7m48.363143054s) [noise=true , address=145.79.227.31:4798 , piece_id=ETAZFJPYJM6Z3VOMBLQIAZNCLZSBLBTVUDQKXLTUPTPKITRYMUBQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.492138500 (624h7m48.363221545s) [noise=true , address=145.79.227.13:4804 , piece_id=UIGCI44TVPTQ5FRKZHJMI4EBSLMSXRA3ULL5NAGQ24G3E7SZ4VUQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.492550854 (624h7m48.363633889s) [noise=true , address=45.91.116.13:4156 , piece_id=K5OLWRAKYGKKEUR6EDFH3UGNPEEF5DXFQTFKR6E7EYCFH6WIH32A, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.492573638 (624h7m48.363656683s) [noise=true , address=145.79.227.39:4014 , piece_id=ECDBG4DPYI5GSWYY5BAAKVISLH7KLGZRNWH6GCDHSFGAB24PLGMA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.493007864 (624h7m48.364090909s) [noise=true , address=77.90.31.17:4773 , piece_id=VDQZE6EUAYTVV3UNTSQJXQK5IE5X4U45NWBW2LK5FQJEYSBECFNA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.493214427 (624h7m48.364297472s) [noise=true , address=145.79.227.48:4302 , piece_id=S7CLPIQQACWBJ4KEVAQV2FCDX5MZ6OLRQEKRRCZOPCB5FPDLYJWA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.493379181 (624h7m48.364462226s) [noise=true , address=45.91.116.9:4261 , piece_id=TYV7MU34ZR36FANU5OKRF7HWVX7CAQVHXQGDT7V5TI4KDYJNSR7A, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.493764325 (624h7m48.36484736s) [noise=true , address=45.91.116.8:4047 , piece_id=FYPOHCLXGXR5SIAC6PVARG33HPMAQEG2FFWFLDHZ4EBWJFVGZROQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.494174055 (624h7m48.3652571s) [noise=true , address=145.79.227.24:4582 , piece_id=SMX5ZM4W6LES6DJGI45FKJSL2QBYWNFQMH2KTJ5YWCZ4QM5C6S4A, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.494592652 (624h7m48.365675687s) [noise=true , address=145.79.227.16:4909 , piece_id=C6AAVPB66FC5WYHW54ZNLU6WQQ7JWXSYKJKLQ5N5JTNF5MNNPICQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:50.198765175 (624h7m50.06984822s) [upload=546917]: Next returned nil segment Dec 31 15:36:50.198809388 (624h7m50.069892424s) [upload=546917]: Waiting for error group managing segments... Dec 31 15:36:50.478858922 (624h7m50.349941957s) [noise=true , address=145.79.227.16:4909 , piece_id=C6AAVPB66FC5WYHW54ZNLU6WQQ7JWXSYKJKLQ5N5JTNF5MNNPICQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.481636421 (624h7m50.352719456s) [noise=true , address=145.79.227.9:4365 , piece_id=MXQ3RZ7RMQIPF56JZ2L5ZRPUQKXUROCFU3I5MF43TO3RLNESAMIQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.483332751 (624h7m50.354415786s) [noise=true , address=145.79.227.25:4619 , piece_id=S5WLABYCZ6VLTRZVEAK7K4RAWOM5EFSSHYAEHRRA6DI6FSRX3XKQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.488309287 (624h7m50.359392332s) [noise=true , address=145.79.227.51:4410 , piece_id=DKXRCAHY3BLW5UNYJLUTDKNMVSMYBRRQPUC4QJ4J64DNNVVHKVQA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.491075704 (624h7m50.362158739s) [noise=true , address=145.79.227.48:4302 , piece_id=S7CLPIQQACWBJ4KEVAQV2FCDX5MZ6OLRQEKRRCZOPCB5FPDLYJWA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.494256071 (624h7m50.365339106s) [noise=true , address=145.79.227.13:4804 , piece_id=UIGCI44TVPTQ5FRKZHJMI4EBSLMSXRA3ULL5NAGQ24G3E7SZ4VUQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.495823866 (624h7m50.366906911s) [noise=true , address=145.79.227.20:4439 , piece_id=AR2RRIAIWIETLYUFBCX6KUFNZ72EYOUCVDV4FWL7OAITLATG3M4Q, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.500817575 (624h7m50.37190061s) [noise=true , address=145.79.227.31:4798 , piece_id=ETAZFJPYJM6Z3VOMBLQIAZNCLZSBLBTVUDQKXLTUPTPKITRYMUBQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.569202799 (624h7m50.440285844s) [noise=true , address=145.79.227.39:4014 , piece_id=ECDBG4DPYI5GSWYY5BAAKVISLH7KLGZRNWH6GCDHSFGAB24PLGMA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.569840523 (624h7m50.440923568s) [noise=true , address=145.79.227.24:4582 , piece_id=SMX5ZM4W6LES6DJGI45FKJSL2QBYWNFQMH2KTJ5YWCZ4QM5C6S4A, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.645228664 (624h7m50.516311709s) [noise=true , address=77.90.31.21:4877 , piece_id=ZX73DOP2WJU3FLD6KBIZBZXK4LWML2NGURRA6QLR7P3BCPFAG3VA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.743535203 (624h7m50.614618248s) [noise=true , address=77.90.31.19:4842 , piece_id=JON75QF3WXYTC6MF34PWDUUUPJ632D3S4TCJF7CNN7ZALIBURAOA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.911323285 (624h7m50.78240632s) [noise=true , address=45.91.116.9:4261 , piece_id=TYV7MU34ZR36FANU5OKRF7HWVX7CAQVHXQGDT7V5TI4KDYJNSR7A, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.911341699 (624h7m50.782424734s) [seg_pos={0 0}, upload=546917]: Segment reached optimal threshold of 13 pieces. Dec 31 15:36:50.911540248 (624h7m50.782623283s) [noise=true , address=45.91.116.8:4047 , piece_id=FYPOHCLXGXR5SIAC6PVARG33HPMAQEG2FFWFLDHZ4EBWJFVGZROQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 31 15:36:50.911545968 (624h7m50.782629003s) [noise=true , address=45.91.116.8:4047 , piece_id=FYPOHCLXGXR5SIAC6PVARG33HPMAQEG2FFWFLDHZ4EBWJFVGZROQ, seg_pos={0 0}, upload=546917]: Long tail cancellation due to optimal threshold Dec 31 15:36:50.911733586 (624h7m50.782816621s) [noise=true , address=77.90.31.17:4773 , piece_id=VDQZE6EUAYTVV3UNTSQJXQK5IE5X4U45NWBW2LK5FQJEYSBECFNA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:59734->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:59734->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59734->77.90.31.17:4773: use of closed network connection Dec 31 15:36:50.911738926 (624h7m50.782821961s) [noise=true , address=77.90.31.17:4773 , piece_id=VDQZE6EUAYTVV3UNTSQJXQK5IE5X4U45NWBW2LK5FQJEYSBECFNA, seg_pos={0 0}, upload=546917]: Long tail cancellation due to optimal threshold Dec 31 15:36:50.911802247 (624h7m50.782885282s) [noise=true , address=45.91.116.13:4156 , piece_id=K5OLWRAKYGKKEUR6EDFH3UGNPEEF5DXFQTFKR6E7EYCFH6WIH32A, seg_pos={0 0}, upload=546917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:48290->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:48290->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48290->45.91.116.13:4156: use of closed network connection Dec 31 15:36:50.911819800 (624h7m50.782902835s) [noise=true , address=45.91.116.13:4156 , piece_id=K5OLWRAKYGKKEUR6EDFH3UGNPEEF5DXFQTFKR6E7EYCFH6WIH32A, seg_pos={0 0}, upload=546917]: Long tail cancellation due to optimal threshold Dec 31 15:36:50.911856039 (624h7m50.782939074s) [seg_pos={0 0}, upload=546917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:36:50.912084494 (624h7m50.783167529s) [upload=546917]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:36:50.912091387 (624h7m50.783174422s) [upload=546917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:36:51.207593394 (624h7m51.078676429s) [upload=546917]: Done uploading segments... Dec 31 15:36:51.207619214 (624h7m51.078702259s) [upload=546917]: Upload finished. err: <nil> Duration: 5.834017147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...X] duration: 30.002440194s trace: https://jaeger.dev.storj.io/trace/2d9b9761b902a2f6 FAIL (2025-12-31 15:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: ZX73DOP2WJU3FLD6KBIZBZXK4LWML2NGURRA6QLR7P3BCPFAG3VA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: JON75QF3WXYTC6MF34PWDUUUPJ632D3S4TCJF7CNN7ZALIBURAOA) context deadline exceeded ======LOGS====== 624h7m48.112592241s) [upload=546917]: Uploading segments... Dec 31 15:36:48.241677396 (624h7m48.112760431s) [upload=546917]: Got next segment. Inline: false Dec 31 15:36:48.241749483 (624h7m48.112832519s) [upload=546917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:36:48.241757859 (624h7m48.112840924s) [upload=546917]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:36:48.490069609 (624h7m48.361152655s) [seg_pos={0 0}, upload=546917]: Begin upload segment... Dec 31 15:36:48.490125506 (624h7m48.361208551s) [seg_pos={0 0}, upload=546917]: Done begin upload segment. Dec 31 15:36:48.490155103 (624h7m48.361238158s) [noise=true , address=145.79.227.20:4439 , piece_id=AR2RRIAIWIETLYUFBCX6KUFNZ72EYOUCVDV4FWL7OAITLATG3M4Q, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.490211570 (624h7m48.361294615s) [noise=true , address=77.90.31.21:4877 , piece_id=ZX73DOP2WJU3FLD6KBIZBZXK4LWML2NGURRA6QLR7P3BCPFAG3VA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.490968261 (624h7m48.362051306s) [noise=true , address=77.90.31.19:4842 , piece_id=JON75QF3WXYTC6MF34PWDUUUPJ632D3S4TCJF7CNN7ZALIBURAOA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.491043384 (624h7m48.362126439s) [noise=true , address=145.79.227.9:4365 , piece_id=MXQ3RZ7RMQIPF56JZ2L5ZRPUQKXUROCFU3I5MF43TO3RLNESAMIQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.491377240 (624h7m48.362460275s) [noise=true , address=145.79.227.51:4410 , piece_id=DKXRCAHY3BLW5UNYJLUTDKNMVSMYBRRQPUC4QJ4J64DNNVVHKVQA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.491757414 (624h7m48.362840469s) [noise=true , address=145.79.227.25:4619 , piece_id=S5WLABYCZ6VLTRZVEAK7K4RAWOM5EFSSHYAEHRRA6DI6FSRX3XKQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.492060009 (624h7m48.363143054s) [noise=true , address=145.79.227.31:4798 , piece_id=ETAZFJPYJM6Z3VOMBLQIAZNCLZSBLBTVUDQKXLTUPTPKITRYMUBQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.492138500 (624h7m48.363221545s) [noise=true , address=145.79.227.13:4804 , piece_id=UIGCI44TVPTQ5FRKZHJMI4EBSLMSXRA3ULL5NAGQ24G3E7SZ4VUQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.492550854 (624h7m48.363633889s) [noise=true , address=45.91.116.13:4156 , piece_id=K5OLWRAKYGKKEUR6EDFH3UGNPEEF5DXFQTFKR6E7EYCFH6WIH32A, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.492573638 (624h7m48.363656683s) [noise=true , address=145.79.227.39:4014 , piece_id=ECDBG4DPYI5GSWYY5BAAKVISLH7KLGZRNWH6GCDHSFGAB24PLGMA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.493007864 (624h7m48.364090909s) [noise=true , address=77.90.31.17:4773 , piece_id=VDQZE6EUAYTVV3UNTSQJXQK5IE5X4U45NWBW2LK5FQJEYSBECFNA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.493214427 (624h7m48.364297472s) [noise=true , address=145.79.227.48:4302 , piece_id=S7CLPIQQACWBJ4KEVAQV2FCDX5MZ6OLRQEKRRCZOPCB5FPDLYJWA, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.493379181 (624h7m48.364462226s) [noise=true , address=45.91.116.9:4261 , piece_id=TYV7MU34ZR36FANU5OKRF7HWVX7CAQVHXQGDT7V5TI4KDYJNSR7A, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.493764325 (624h7m48.36484736s) [noise=true , address=45.91.116.8:4047 , piece_id=FYPOHCLXGXR5SIAC6PVARG33HPMAQEG2FFWFLDHZ4EBWJFVGZROQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.494174055 (624h7m48.3652571s) [noise=true , address=145.79.227.24:4582 , piece_id=SMX5ZM4W6LES6DJGI45FKJSL2QBYWNFQMH2KTJ5YWCZ4QM5C6S4A, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:48.494592652 (624h7m48.365675687s) [noise=true , address=145.79.227.16:4909 , piece_id=C6AAVPB66FC5WYHW54ZNLU6WQQ7JWXSYKJKLQ5N5JTNF5MNNPICQ, seg_pos={0 0}, upload=546917]: Uploading piece... Dec 31 15:36:50.198765175 (624h7m50.06984822s) [upload=546917]: Next returned nil segment Dec 31 15:36:50.198809388 (624h7m50.069892424s) [upload=546917]: Waiting for error group managing segments... Dec 31 15:36:50.478858922 (624h7m50.349941957s) [noise=true , address=145.79.227.16:4909 , piece_id=C6AAVPB66FC5WYHW54ZNLU6WQQ7JWXSYKJKLQ5N5JTNF5MNNPICQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.481636421 (624h7m50.352719456s) [noise=true , address=145.79.227.9:4365 , piece_id=MXQ3RZ7RMQIPF56JZ2L5ZRPUQKXUROCFU3I5MF43TO3RLNESAMIQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.483332751 (624h7m50.354415786s) [noise=true , address=145.79.227.25:4619 , piece_id=S5WLABYCZ6VLTRZVEAK7K4RAWOM5EFSSHYAEHRRA6DI6FSRX3XKQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.488309287 (624h7m50.359392332s) [noise=true , address=145.79.227.51:4410 , piece_id=DKXRCAHY3BLW5UNYJLUTDKNMVSMYBRRQPUC4QJ4J64DNNVVHKVQA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.491075704 (624h7m50.362158739s) [noise=true , address=145.79.227.48:4302 , piece_id=S7CLPIQQACWBJ4KEVAQV2FCDX5MZ6OLRQEKRRCZOPCB5FPDLYJWA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.494256071 (624h7m50.365339106s) [noise=true , address=145.79.227.13:4804 , piece_id=UIGCI44TVPTQ5FRKZHJMI4EBSLMSXRA3ULL5NAGQ24G3E7SZ4VUQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.495823866 (624h7m50.366906911s) [noise=true , address=145.79.227.20:4439 , piece_id=AR2RRIAIWIETLYUFBCX6KUFNZ72EYOUCVDV4FWL7OAITLATG3M4Q, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.500817575 (624h7m50.37190061s) [noise=true , address=145.79.227.31:4798 , piece_id=ETAZFJPYJM6Z3VOMBLQIAZNCLZSBLBTVUDQKXLTUPTPKITRYMUBQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.569202799 (624h7m50.440285844s) [noise=true , address=145.79.227.39:4014 , piece_id=ECDBG4DPYI5GSWYY5BAAKVISLH7KLGZRNWH6GCDHSFGAB24PLGMA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.569840523 (624h7m50.440923568s) [noise=true , address=145.79.227.24:4582 , piece_id=SMX5ZM4W6LES6DJGI45FKJSL2QBYWNFQMH2KTJ5YWCZ4QM5C6S4A, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.645228664 (624h7m50.516311709s) [noise=true , address=77.90.31.21:4877 , piece_id=ZX73DOP2WJU3FLD6KBIZBZXK4LWML2NGURRA6QLR7P3BCPFAG3VA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.743535203 (624h7m50.614618248s) [noise=true , address=77.90.31.19:4842 , piece_id=JON75QF3WXYTC6MF34PWDUUUPJ632D3S4TCJF7CNN7ZALIBURAOA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.911323285 (624h7m50.78240632s) [noise=true , address=45.91.116.9:4261 , piece_id=TYV7MU34ZR36FANU5OKRF7HWVX7CAQVHXQGDT7V5TI4KDYJNSR7A, seg_pos={0 0}, upload=546917]: Done uploading piece. err: <nil> Dec 31 15:36:50.911341699 (624h7m50.782424734s) [seg_pos={0 0}, upload=546917]: Segment reached optimal threshold of 13 pieces. Dec 31 15:36:50.911540248 (624h7m50.782623283s) [noise=true , address=45.91.116.8:4047 , piece_id=FYPOHCLXGXR5SIAC6PVARG33HPMAQEG2FFWFLDHZ4EBWJFVGZROQ, seg_pos={0 0}, upload=546917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 31 15:36:50.911545968 (624h7m50.782629003s) [noise=true , address=45.91.116.8:4047 , piece_id=FYPOHCLXGXR5SIAC6PVARG33HPMAQEG2FFWFLDHZ4EBWJFVGZROQ, seg_pos={0 0}, upload=546917]: Long tail cancellation due to optimal threshold Dec 31 15:36:50.911733586 (624h7m50.782816621s) [noise=true , address=77.90.31.17:4773 , piece_id=VDQZE6EUAYTVV3UNTSQJXQK5IE5X4U45NWBW2LK5FQJEYSBECFNA, seg_pos={0 0}, upload=546917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:59734->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:59734->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59734->77.90.31.17:4773: use of closed network connection Dec 31 15:36:50.911738926 (624h7m50.782821961s) [noise=true , address=77.90.31.17:4773 , piece_id=VDQZE6EUAYTVV3UNTSQJXQK5IE5X4U45NWBW2LK5FQJEYSBECFNA, seg_pos={0 0}, upload=546917]: Long tail cancellation due to optimal threshold Dec 31 15:36:50.911802247 (624h7m50.782885282s) [noise=true , address=45.91.116.13:4156 , piece_id=K5OLWRAKYGKKEUR6EDFH3UGNPEEF5DXFQTFKR6E7EYCFH6WIH32A, seg_pos={0 0}, upload=546917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:48290->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:48290->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48290->45.91.116.13:4156: use of closed network connection Dec 31 15:36:50.911819800 (624h7m50.782902835s) [noise=true , address=45.91.116.13:4156 , piece_id=K5OLWRAKYGKKEUR6EDFH3UGNPEEF5DXFQTFKR6E7EYCFH6WIH32A, seg_pos={0 0}, upload=546917]: Long tail cancellation due to optimal threshold Dec 31 15:36:50.911856039 (624h7m50.782939074s) [seg_pos={0 0}, upload=546917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:36:50.912084494 (624h7m50.783167529s) [upload=546917]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:36:50.912091387 (624h7m50.783174422s) [upload=546917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:36:51.207593394 (624h7m51.078676429s) [upload=546917]: Done uploading segments... Dec 31 15:36:51.207619214 (624h7m51.078702259s) [upload=546917]: Upload finished. err: <nil> Duration: 30.002459139s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...........X..XX...........XXX..X.XX..XX..XXXXX...X.X.X.....] duration: 3.88202607s trace: https://jaeger.dev.storj.io/trace/1901bc26af4982d6 FAIL (2025-12-31 15:36:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.882061799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.X.........X.....................] duration: 30.746494815s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:36:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 30.746519343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX...] duration: 5.995192251s trace: https://jaeger.dev.storj.io/trace/3c39c9eda59a9b7a FAIL (2025-12-31 15:36:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA) context deadline exceeded ======LOGS====== ad=546880]: Got next segment. Inline: false Dec 31 15:34:22.650227351 (624h5m22.521310396s) [upload=546880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:34:22.650235517 (624h5m22.521318572s) [upload=546880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:34:22.896810300 (624h5m22.767893335s) [seg_pos={0 0}, upload=546880]: Begin upload segment... Dec 31 15:34:22.896855636 (624h5m22.767938681s) [seg_pos={0 0}, upload=546880]: Done begin upload segment. Dec 31 15:34:22.896889901 (624h5m22.767972976s) [noise=true , address=45.91.116.6:4975 , piece_id=L5VIAMA35SD6VHWO4VX7CFVI3XXVX2YYYVEITXMKWRG22N2XXOOQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.897069413 (624h5m22.768152448s) [noise=true , address=145.79.227.47:4266 , piece_id=IEXPATPPCHXO3D3KF5576BAWQCON5H4Q24PDEH5E5MYCOMJ74EKA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.897587319 (624h5m22.768670364s) [noise=true , address=145.79.227.51:4409 , piece_id=7HKM6WZKBXJM236G72E2RAY2ZRUHEVQ7GTAUM7Q73A3EVHVQLKJQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898023691 (624h5m22.769106736s) [noise=true , address=145.79.227.11:4731 , piece_id=HZQOJAVTWSSDEOATABCCHS7ZNOI3EHELQ7EGXC6V2GIWIAZB25CQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898481322 (624h5m22.769564367s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898969361 (624h5m22.770052406s) [noise=true , address=145.79.227.26:4654 , piece_id=OGHN3XLLYGTFXKEHMFDOYBAWWI43KWUYSJCBVGLG5XDGM3L5UXSQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.899419989 (624h5m22.770503024s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.899809671 (624h5m22.770892716s) [noise=true , address=145.79.227.46:4602 , piece_id=CQDLHIA4XYUC6FEJBLFI33IDC6INU6PK2HYSCCKEPXYOW4SYGTIQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.900236564 (624h5m22.771319599s) [noise=true , address=145.79.227.2:4114 , piece_id=VVYXESPVLX5MZLVCXIOHDQRTO4MGW7FA6FRFBZ2XGORPD5ZKCGJA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.900593643 (624h5m22.771676688s) [noise=true , address=145.79.227.39:4014 , piece_id=FBZKHYOOGMJJZUNRL7PCFQR7LMDBNYAZSWXA2YDSM77JDBFBTCYQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.905994818 (624h5m22.777077863s) [noise=true , address=145.79.227.15:4876 , piece_id=KR3GVSCRWATOYQXO5NWRJYMLZ5YSSPWGBQ7LX2DNLBXRJX77GNAA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.907443236 (624h5m22.778526271s) [noise=true , address=145.79.227.18:4007 , piece_id=X4LYOGGBCAVQUW4MHO6TNTQLTCQ676TE4IKXK7PZCRJZRTLXA3ZA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.910908795 (624h5m22.78199184s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.912878255 (624h5m22.783961301s) [noise=true , address=45.91.116.11:4118 , piece_id=J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.914854348 (624h5m22.785937393s) [noise=true , address=45.91.116.8:4046 , piece_id=6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.915350071 (624h5m22.786433116s) [noise=true , address=145.79.227.37:4567 , piece_id=UR7S2CVPB5O2QOAGV6FLYABXVBD3Z274J5NEHIQT2YIINMJ37GAQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:24.613334152 (624h5m24.484417197s) [upload=546880]: Next returned nil segment Dec 31 15:34:24.613348319 (624h5m24.484431354s) [upload=546880]: Waiting for error group managing segments... Dec 31 15:34:24.894856509 (624h5m24.765939545s) [noise=true , address=145.79.227.46:4602 , piece_id=CQDLHIA4XYUC6FEJBLFI33IDC6INU6PK2HYSCCKEPXYOW4SYGTIQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.897734480 (624h5m24.768817515s) [noise=true , address=145.79.227.37:4567 , piece_id=UR7S2CVPB5O2QOAGV6FLYABXVBD3Z274J5NEHIQT2YIINMJ37GAQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.906390712 (624h5m24.777473767s) [noise=true , address=145.79.227.11:4731 , piece_id=HZQOJAVTWSSDEOATABCCHS7ZNOI3EHELQ7EGXC6V2GIWIAZB25CQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.909468052 (624h5m24.780551097s) [noise=true , address=145.79.227.18:4007 , piece_id=X4LYOGGBCAVQUW4MHO6TNTQLTCQ676TE4IKXK7PZCRJZRTLXA3ZA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.915932321 (624h5m24.787015366s) [noise=true , address=145.79.227.15:4876 , piece_id=KR3GVSCRWATOYQXO5NWRJYMLZ5YSSPWGBQ7LX2DNLBXRJX77GNAA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.938445078 (624h5m24.809528124s) [noise=true , address=145.79.227.47:4266 , piece_id=IEXPATPPCHXO3D3KF5576BAWQCON5H4Q24PDEH5E5MYCOMJ74EKA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.969310681 (624h5m24.840393726s) [noise=true , address=145.79.227.2:4114 , piece_id=VVYXESPVLX5MZLVCXIOHDQRTO4MGW7FA6FRFBZ2XGORPD5ZKCGJA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.974435920 (624h5m24.845518965s) [noise=true , address=145.79.227.39:4014 , piece_id=FBZKHYOOGMJJZUNRL7PCFQR7LMDBNYAZSWXA2YDSM77JDBFBTCYQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.976432863 (624h5m24.847515898s) [noise=true , address=145.79.227.26:4654 , piece_id=OGHN3XLLYGTFXKEHMFDOYBAWWI43KWUYSJCBVGLG5XDGM3L5UXSQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.990945277 (624h5m24.862028322s) [noise=true , address=145.79.227.51:4409 , piece_id=7HKM6WZKBXJM236G72E2RAY2ZRUHEVQ7GTAUM7Q73A3EVHVQLKJQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.238863136 (624h5m25.109946171s) [noise=true , address=45.91.116.11:4118 , piece_id=J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.254067348 (624h5m25.125150403s) [noise=true , address=45.91.116.6:4975 , piece_id=L5VIAMA35SD6VHWO4VX7CFVI3XXVX2YYYVEITXMKWRG22N2XXOOQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.290212573 (624h5m25.161295629s) [noise=true , address=45.91.116.8:4046 , piece_id=6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.290259833 (624h5m25.161342888s) [seg_pos={0 0}, upload=546880]: Segment reached optimal threshold of 13 pieces. Dec 31 15:34:25.290501563 (624h5m25.161584618s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection Dec 31 15:34:25.290509138 (624h5m25.161592173s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290531440 (624h5m25.161614475s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection Dec 31 15:34:25.290539726 (624h5m25.161622771s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290872250 (624h5m25.161955285s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: read tcp 188.166.225.170:41686->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 31 15:34:25.290878822 (624h5m25.161961867s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290895042 (624h5m25.161978077s) [seg_pos={0 0}, upload=546880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:34:25.290990114 (624h5m25.162073159s) [upload=546880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:34:25.290995724 (624h5m25.162078759s) [upload=546880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:34:25.591032756 (624h5m25.462115791s) [upload=546880]: Done uploading segments... Dec 31 15:34:25.591049088 (624h5m25.462132123s) [upload=546880]: Upload finished. err: <nil> Duration: 5.995211528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.X.........X....................] duration: 33.869706334s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:35:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 33.869726372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX..] duration: 5.866617042s trace: https://jaeger.dev.storj.io/trace/3c39c9eda59a9b7a FAIL (2025-12-31 15:35:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA) context deadline exceeded ======LOGS====== ad=546880]: Got next segment. Inline: false Dec 31 15:34:22.650227351 (624h5m22.521310396s) [upload=546880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:34:22.650235517 (624h5m22.521318572s) [upload=546880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:34:22.896810300 (624h5m22.767893335s) [seg_pos={0 0}, upload=546880]: Begin upload segment... Dec 31 15:34:22.896855636 (624h5m22.767938681s) [seg_pos={0 0}, upload=546880]: Done begin upload segment. Dec 31 15:34:22.896889901 (624h5m22.767972976s) [noise=true , address=45.91.116.6:4975 , piece_id=L5VIAMA35SD6VHWO4VX7CFVI3XXVX2YYYVEITXMKWRG22N2XXOOQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.897069413 (624h5m22.768152448s) [noise=true , address=145.79.227.47:4266 , piece_id=IEXPATPPCHXO3D3KF5576BAWQCON5H4Q24PDEH5E5MYCOMJ74EKA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.897587319 (624h5m22.768670364s) [noise=true , address=145.79.227.51:4409 , piece_id=7HKM6WZKBXJM236G72E2RAY2ZRUHEVQ7GTAUM7Q73A3EVHVQLKJQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898023691 (624h5m22.769106736s) [noise=true , address=145.79.227.11:4731 , piece_id=HZQOJAVTWSSDEOATABCCHS7ZNOI3EHELQ7EGXC6V2GIWIAZB25CQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898481322 (624h5m22.769564367s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898969361 (624h5m22.770052406s) [noise=true , address=145.79.227.26:4654 , piece_id=OGHN3XLLYGTFXKEHMFDOYBAWWI43KWUYSJCBVGLG5XDGM3L5UXSQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.899419989 (624h5m22.770503024s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.899809671 (624h5m22.770892716s) [noise=true , address=145.79.227.46:4602 , piece_id=CQDLHIA4XYUC6FEJBLFI33IDC6INU6PK2HYSCCKEPXYOW4SYGTIQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.900236564 (624h5m22.771319599s) [noise=true , address=145.79.227.2:4114 , piece_id=VVYXESPVLX5MZLVCXIOHDQRTO4MGW7FA6FRFBZ2XGORPD5ZKCGJA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.900593643 (624h5m22.771676688s) [noise=true , address=145.79.227.39:4014 , piece_id=FBZKHYOOGMJJZUNRL7PCFQR7LMDBNYAZSWXA2YDSM77JDBFBTCYQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.905994818 (624h5m22.777077863s) [noise=true , address=145.79.227.15:4876 , piece_id=KR3GVSCRWATOYQXO5NWRJYMLZ5YSSPWGBQ7LX2DNLBXRJX77GNAA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.907443236 (624h5m22.778526271s) [noise=true , address=145.79.227.18:4007 , piece_id=X4LYOGGBCAVQUW4MHO6TNTQLTCQ676TE4IKXK7PZCRJZRTLXA3ZA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.910908795 (624h5m22.78199184s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.912878255 (624h5m22.783961301s) [noise=true , address=45.91.116.11:4118 , piece_id=J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.914854348 (624h5m22.785937393s) [noise=true , address=45.91.116.8:4046 , piece_id=6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.915350071 (624h5m22.786433116s) [noise=true , address=145.79.227.37:4567 , piece_id=UR7S2CVPB5O2QOAGV6FLYABXVBD3Z274J5NEHIQT2YIINMJ37GAQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:24.613334152 (624h5m24.484417197s) [upload=546880]: Next returned nil segment Dec 31 15:34:24.613348319 (624h5m24.484431354s) [upload=546880]: Waiting for error group managing segments... Dec 31 15:34:24.894856509 (624h5m24.765939545s) [noise=true , address=145.79.227.46:4602 , piece_id=CQDLHIA4XYUC6FEJBLFI33IDC6INU6PK2HYSCCKEPXYOW4SYGTIQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.897734480 (624h5m24.768817515s) [noise=true , address=145.79.227.37:4567 , piece_id=UR7S2CVPB5O2QOAGV6FLYABXVBD3Z274J5NEHIQT2YIINMJ37GAQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.906390712 (624h5m24.777473767s) [noise=true , address=145.79.227.11:4731 , piece_id=HZQOJAVTWSSDEOATABCCHS7ZNOI3EHELQ7EGXC6V2GIWIAZB25CQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.909468052 (624h5m24.780551097s) [noise=true , address=145.79.227.18:4007 , piece_id=X4LYOGGBCAVQUW4MHO6TNTQLTCQ676TE4IKXK7PZCRJZRTLXA3ZA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.915932321 (624h5m24.787015366s) [noise=true , address=145.79.227.15:4876 , piece_id=KR3GVSCRWATOYQXO5NWRJYMLZ5YSSPWGBQ7LX2DNLBXRJX77GNAA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.938445078 (624h5m24.809528124s) [noise=true , address=145.79.227.47:4266 , piece_id=IEXPATPPCHXO3D3KF5576BAWQCON5H4Q24PDEH5E5MYCOMJ74EKA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.969310681 (624h5m24.840393726s) [noise=true , address=145.79.227.2:4114 , piece_id=VVYXESPVLX5MZLVCXIOHDQRTO4MGW7FA6FRFBZ2XGORPD5ZKCGJA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.974435920 (624h5m24.845518965s) [noise=true , address=145.79.227.39:4014 , piece_id=FBZKHYOOGMJJZUNRL7PCFQR7LMDBNYAZSWXA2YDSM77JDBFBTCYQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.976432863 (624h5m24.847515898s) [noise=true , address=145.79.227.26:4654 , piece_id=OGHN3XLLYGTFXKEHMFDOYBAWWI43KWUYSJCBVGLG5XDGM3L5UXSQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.990945277 (624h5m24.862028322s) [noise=true , address=145.79.227.51:4409 , piece_id=7HKM6WZKBXJM236G72E2RAY2ZRUHEVQ7GTAUM7Q73A3EVHVQLKJQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.238863136 (624h5m25.109946171s) [noise=true , address=45.91.116.11:4118 , piece_id=J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.254067348 (624h5m25.125150403s) [noise=true , address=45.91.116.6:4975 , piece_id=L5VIAMA35SD6VHWO4VX7CFVI3XXVX2YYYVEITXMKWRG22N2XXOOQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.290212573 (624h5m25.161295629s) [noise=true , address=45.91.116.8:4046 , piece_id=6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.290259833 (624h5m25.161342888s) [seg_pos={0 0}, upload=546880]: Segment reached optimal threshold of 13 pieces. Dec 31 15:34:25.290501563 (624h5m25.161584618s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection Dec 31 15:34:25.290509138 (624h5m25.161592173s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290531440 (624h5m25.161614475s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection Dec 31 15:34:25.290539726 (624h5m25.161622771s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290872250 (624h5m25.161955285s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: read tcp 188.166.225.170:41686->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 31 15:34:25.290878822 (624h5m25.161961867s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290895042 (624h5m25.161978077s) [seg_pos={0 0}, upload=546880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:34:25.290990114 (624h5m25.162073159s) [upload=546880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:34:25.290995724 (624h5m25.162078759s) [upload=546880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:34:25.591032756 (624h5m25.462115791s) [upload=546880]: Done uploading segments... Dec 31 15:34:25.591049088 (624h5m25.462132123s) [upload=546880]: Upload finished. err: <nil> Duration: 5.866673769s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [............X..XX...........XXX..X.XX..XX..XXXXX...X.X.X....] duration: 3.676980985s trace: https://jaeger.dev.storj.io/trace/1901bc26af4982d6 FAIL (2025-12-31 15:35:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.677005721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.X.........X...................] duration: 33.590529754s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:35:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 33.590558648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX.] duration: 6.123137372s trace: https://jaeger.dev.storj.io/trace/3c39c9eda59a9b7a FAIL (2025-12-31 15:34:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA) context deadline exceeded ======LOGS====== ad=546880]: Got next segment. Inline: false Dec 31 15:34:22.650227351 (624h5m22.521310396s) [upload=546880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:34:22.650235517 (624h5m22.521318572s) [upload=546880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:34:22.896810300 (624h5m22.767893335s) [seg_pos={0 0}, upload=546880]: Begin upload segment... Dec 31 15:34:22.896855636 (624h5m22.767938681s) [seg_pos={0 0}, upload=546880]: Done begin upload segment. Dec 31 15:34:22.896889901 (624h5m22.767972976s) [noise=true , address=45.91.116.6:4975 , piece_id=L5VIAMA35SD6VHWO4VX7CFVI3XXVX2YYYVEITXMKWRG22N2XXOOQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.897069413 (624h5m22.768152448s) [noise=true , address=145.79.227.47:4266 , piece_id=IEXPATPPCHXO3D3KF5576BAWQCON5H4Q24PDEH5E5MYCOMJ74EKA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.897587319 (624h5m22.768670364s) [noise=true , address=145.79.227.51:4409 , piece_id=7HKM6WZKBXJM236G72E2RAY2ZRUHEVQ7GTAUM7Q73A3EVHVQLKJQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898023691 (624h5m22.769106736s) [noise=true , address=145.79.227.11:4731 , piece_id=HZQOJAVTWSSDEOATABCCHS7ZNOI3EHELQ7EGXC6V2GIWIAZB25CQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898481322 (624h5m22.769564367s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898969361 (624h5m22.770052406s) [noise=true , address=145.79.227.26:4654 , piece_id=OGHN3XLLYGTFXKEHMFDOYBAWWI43KWUYSJCBVGLG5XDGM3L5UXSQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.899419989 (624h5m22.770503024s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.899809671 (624h5m22.770892716s) [noise=true , address=145.79.227.46:4602 , piece_id=CQDLHIA4XYUC6FEJBLFI33IDC6INU6PK2HYSCCKEPXYOW4SYGTIQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.900236564 (624h5m22.771319599s) [noise=true , address=145.79.227.2:4114 , piece_id=VVYXESPVLX5MZLVCXIOHDQRTO4MGW7FA6FRFBZ2XGORPD5ZKCGJA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.900593643 (624h5m22.771676688s) [noise=true , address=145.79.227.39:4014 , piece_id=FBZKHYOOGMJJZUNRL7PCFQR7LMDBNYAZSWXA2YDSM77JDBFBTCYQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.905994818 (624h5m22.777077863s) [noise=true , address=145.79.227.15:4876 , piece_id=KR3GVSCRWATOYQXO5NWRJYMLZ5YSSPWGBQ7LX2DNLBXRJX77GNAA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.907443236 (624h5m22.778526271s) [noise=true , address=145.79.227.18:4007 , piece_id=X4LYOGGBCAVQUW4MHO6TNTQLTCQ676TE4IKXK7PZCRJZRTLXA3ZA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.910908795 (624h5m22.78199184s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.912878255 (624h5m22.783961301s) [noise=true , address=45.91.116.11:4118 , piece_id=J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.914854348 (624h5m22.785937393s) [noise=true , address=45.91.116.8:4046 , piece_id=6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.915350071 (624h5m22.786433116s) [noise=true , address=145.79.227.37:4567 , piece_id=UR7S2CVPB5O2QOAGV6FLYABXVBD3Z274J5NEHIQT2YIINMJ37GAQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:24.613334152 (624h5m24.484417197s) [upload=546880]: Next returned nil segment Dec 31 15:34:24.613348319 (624h5m24.484431354s) [upload=546880]: Waiting for error group managing segments... Dec 31 15:34:24.894856509 (624h5m24.765939545s) [noise=true , address=145.79.227.46:4602 , piece_id=CQDLHIA4XYUC6FEJBLFI33IDC6INU6PK2HYSCCKEPXYOW4SYGTIQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.897734480 (624h5m24.768817515s) [noise=true , address=145.79.227.37:4567 , piece_id=UR7S2CVPB5O2QOAGV6FLYABXVBD3Z274J5NEHIQT2YIINMJ37GAQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.906390712 (624h5m24.777473767s) [noise=true , address=145.79.227.11:4731 , piece_id=HZQOJAVTWSSDEOATABCCHS7ZNOI3EHELQ7EGXC6V2GIWIAZB25CQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.909468052 (624h5m24.780551097s) [noise=true , address=145.79.227.18:4007 , piece_id=X4LYOGGBCAVQUW4MHO6TNTQLTCQ676TE4IKXK7PZCRJZRTLXA3ZA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.915932321 (624h5m24.787015366s) [noise=true , address=145.79.227.15:4876 , piece_id=KR3GVSCRWATOYQXO5NWRJYMLZ5YSSPWGBQ7LX2DNLBXRJX77GNAA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.938445078 (624h5m24.809528124s) [noise=true , address=145.79.227.47:4266 , piece_id=IEXPATPPCHXO3D3KF5576BAWQCON5H4Q24PDEH5E5MYCOMJ74EKA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.969310681 (624h5m24.840393726s) [noise=true , address=145.79.227.2:4114 , piece_id=VVYXESPVLX5MZLVCXIOHDQRTO4MGW7FA6FRFBZ2XGORPD5ZKCGJA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.974435920 (624h5m24.845518965s) [noise=true , address=145.79.227.39:4014 , piece_id=FBZKHYOOGMJJZUNRL7PCFQR7LMDBNYAZSWXA2YDSM77JDBFBTCYQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.976432863 (624h5m24.847515898s) [noise=true , address=145.79.227.26:4654 , piece_id=OGHN3XLLYGTFXKEHMFDOYBAWWI43KWUYSJCBVGLG5XDGM3L5UXSQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.990945277 (624h5m24.862028322s) [noise=true , address=145.79.227.51:4409 , piece_id=7HKM6WZKBXJM236G72E2RAY2ZRUHEVQ7GTAUM7Q73A3EVHVQLKJQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.238863136 (624h5m25.109946171s) [noise=true , address=45.91.116.11:4118 , piece_id=J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.254067348 (624h5m25.125150403s) [noise=true , address=45.91.116.6:4975 , piece_id=L5VIAMA35SD6VHWO4VX7CFVI3XXVX2YYYVEITXMKWRG22N2XXOOQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.290212573 (624h5m25.161295629s) [noise=true , address=45.91.116.8:4046 , piece_id=6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.290259833 (624h5m25.161342888s) [seg_pos={0 0}, upload=546880]: Segment reached optimal threshold of 13 pieces. Dec 31 15:34:25.290501563 (624h5m25.161584618s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection Dec 31 15:34:25.290509138 (624h5m25.161592173s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290531440 (624h5m25.161614475s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection Dec 31 15:34:25.290539726 (624h5m25.161622771s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290872250 (624h5m25.161955285s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: read tcp 188.166.225.170:41686->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 31 15:34:25.290878822 (624h5m25.161961867s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290895042 (624h5m25.161978077s) [seg_pos={0 0}, upload=546880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:34:25.290990114 (624h5m25.162073159s) [upload=546880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:34:25.290995724 (624h5m25.162078759s) [upload=546880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:34:25.591032756 (624h5m25.462115791s) [upload=546880]: Done uploading segments... Dec 31 15:34:25.591049088 (624h5m25.462132123s) [upload=546880]: Upload finished. err: <nil> Duration: 6.123156479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X..XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..XX] duration: 30.003577448s trace: https://jaeger.dev.storj.io/trace/3c39c9eda59a9b7a FAIL (2025-12-31 15:34:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA) context deadline exceeded ======LOGS====== ad=546880]: Got next segment. Inline: false Dec 31 15:34:22.650227351 (624h5m22.521310396s) [upload=546880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:34:22.650235517 (624h5m22.521318572s) [upload=546880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:34:22.896810300 (624h5m22.767893335s) [seg_pos={0 0}, upload=546880]: Begin upload segment... Dec 31 15:34:22.896855636 (624h5m22.767938681s) [seg_pos={0 0}, upload=546880]: Done begin upload segment. Dec 31 15:34:22.896889901 (624h5m22.767972976s) [noise=true , address=45.91.116.6:4975 , piece_id=L5VIAMA35SD6VHWO4VX7CFVI3XXVX2YYYVEITXMKWRG22N2XXOOQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.897069413 (624h5m22.768152448s) [noise=true , address=145.79.227.47:4266 , piece_id=IEXPATPPCHXO3D3KF5576BAWQCON5H4Q24PDEH5E5MYCOMJ74EKA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.897587319 (624h5m22.768670364s) [noise=true , address=145.79.227.51:4409 , piece_id=7HKM6WZKBXJM236G72E2RAY2ZRUHEVQ7GTAUM7Q73A3EVHVQLKJQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898023691 (624h5m22.769106736s) [noise=true , address=145.79.227.11:4731 , piece_id=HZQOJAVTWSSDEOATABCCHS7ZNOI3EHELQ7EGXC6V2GIWIAZB25CQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898481322 (624h5m22.769564367s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.898969361 (624h5m22.770052406s) [noise=true , address=145.79.227.26:4654 , piece_id=OGHN3XLLYGTFXKEHMFDOYBAWWI43KWUYSJCBVGLG5XDGM3L5UXSQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.899419989 (624h5m22.770503024s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.899809671 (624h5m22.770892716s) [noise=true , address=145.79.227.46:4602 , piece_id=CQDLHIA4XYUC6FEJBLFI33IDC6INU6PK2HYSCCKEPXYOW4SYGTIQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.900236564 (624h5m22.771319599s) [noise=true , address=145.79.227.2:4114 , piece_id=VVYXESPVLX5MZLVCXIOHDQRTO4MGW7FA6FRFBZ2XGORPD5ZKCGJA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.900593643 (624h5m22.771676688s) [noise=true , address=145.79.227.39:4014 , piece_id=FBZKHYOOGMJJZUNRL7PCFQR7LMDBNYAZSWXA2YDSM77JDBFBTCYQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.905994818 (624h5m22.777077863s) [noise=true , address=145.79.227.15:4876 , piece_id=KR3GVSCRWATOYQXO5NWRJYMLZ5YSSPWGBQ7LX2DNLBXRJX77GNAA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.907443236 (624h5m22.778526271s) [noise=true , address=145.79.227.18:4007 , piece_id=X4LYOGGBCAVQUW4MHO6TNTQLTCQ676TE4IKXK7PZCRJZRTLXA3ZA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.910908795 (624h5m22.78199184s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.912878255 (624h5m22.783961301s) [noise=true , address=45.91.116.11:4118 , piece_id=J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.914854348 (624h5m22.785937393s) [noise=true , address=45.91.116.8:4046 , piece_id=6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:22.915350071 (624h5m22.786433116s) [noise=true , address=145.79.227.37:4567 , piece_id=UR7S2CVPB5O2QOAGV6FLYABXVBD3Z274J5NEHIQT2YIINMJ37GAQ, seg_pos={0 0}, upload=546880]: Uploading piece... Dec 31 15:34:24.613334152 (624h5m24.484417197s) [upload=546880]: Next returned nil segment Dec 31 15:34:24.613348319 (624h5m24.484431354s) [upload=546880]: Waiting for error group managing segments... Dec 31 15:34:24.894856509 (624h5m24.765939545s) [noise=true , address=145.79.227.46:4602 , piece_id=CQDLHIA4XYUC6FEJBLFI33IDC6INU6PK2HYSCCKEPXYOW4SYGTIQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.897734480 (624h5m24.768817515s) [noise=true , address=145.79.227.37:4567 , piece_id=UR7S2CVPB5O2QOAGV6FLYABXVBD3Z274J5NEHIQT2YIINMJ37GAQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.906390712 (624h5m24.777473767s) [noise=true , address=145.79.227.11:4731 , piece_id=HZQOJAVTWSSDEOATABCCHS7ZNOI3EHELQ7EGXC6V2GIWIAZB25CQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.909468052 (624h5m24.780551097s) [noise=true , address=145.79.227.18:4007 , piece_id=X4LYOGGBCAVQUW4MHO6TNTQLTCQ676TE4IKXK7PZCRJZRTLXA3ZA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.915932321 (624h5m24.787015366s) [noise=true , address=145.79.227.15:4876 , piece_id=KR3GVSCRWATOYQXO5NWRJYMLZ5YSSPWGBQ7LX2DNLBXRJX77GNAA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.938445078 (624h5m24.809528124s) [noise=true , address=145.79.227.47:4266 , piece_id=IEXPATPPCHXO3D3KF5576BAWQCON5H4Q24PDEH5E5MYCOMJ74EKA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.969310681 (624h5m24.840393726s) [noise=true , address=145.79.227.2:4114 , piece_id=VVYXESPVLX5MZLVCXIOHDQRTO4MGW7FA6FRFBZ2XGORPD5ZKCGJA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.974435920 (624h5m24.845518965s) [noise=true , address=145.79.227.39:4014 , piece_id=FBZKHYOOGMJJZUNRL7PCFQR7LMDBNYAZSWXA2YDSM77JDBFBTCYQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.976432863 (624h5m24.847515898s) [noise=true , address=145.79.227.26:4654 , piece_id=OGHN3XLLYGTFXKEHMFDOYBAWWI43KWUYSJCBVGLG5XDGM3L5UXSQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:24.990945277 (624h5m24.862028322s) [noise=true , address=145.79.227.51:4409 , piece_id=7HKM6WZKBXJM236G72E2RAY2ZRUHEVQ7GTAUM7Q73A3EVHVQLKJQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.238863136 (624h5m25.109946171s) [noise=true , address=45.91.116.11:4118 , piece_id=J3KWPGDVL5E7BH5XJOQSR5Z5PK7TFGFU6K22EFWTEXJMBJF3A5UA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.254067348 (624h5m25.125150403s) [noise=true , address=45.91.116.6:4975 , piece_id=L5VIAMA35SD6VHWO4VX7CFVI3XXVX2YYYVEITXMKWRG22N2XXOOQ, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.290212573 (624h5m25.161295629s) [noise=true , address=45.91.116.8:4046 , piece_id=6H7JPK2GR2GUADBBPNATHCK6ZKYHWVILPGQKRI7ORCCILTI5H2QA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: <nil> Dec 31 15:34:25.290259833 (624h5m25.161342888s) [seg_pos={0 0}, upload=546880]: Segment reached optimal threshold of 13 pieces. Dec 31 15:34:25.290501563 (624h5m25.161584618s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40116->45.91.116.14:4189: use of closed network connection Dec 31 15:34:25.290509138 (624h5m25.161592173s) [noise=true , address=45.91.116.14:4189 , piece_id=NDREHLKJGBH7ATALNQ4Y4D3AXVVK42DFJ3V4G3BVBEJSN5DXWGJA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290531440 (624h5m25.161614475s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44834->45.91.116.10:4082: use of closed network connection Dec 31 15:34:25.290539726 (624h5m25.161622771s) [noise=true , address=45.91.116.10:4082 , piece_id=MGTMLECNAE5RZKQYNNN5IA23EDS5TD35JO62NLNXGECTQTJ2CQSA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290872250 (624h5m25.161955285s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: read tcp 188.166.225.170:41686->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 31 15:34:25.290878822 (624h5m25.161961867s) [noise=true , address=77.90.31.13:4629 , piece_id=MIWLAKDI3S6BQJZMGVZSWMUYWEX3GBAZNG5QY725WY6HCP3WH5IA, seg_pos={0 0}, upload=546880]: Long tail cancellation due to optimal threshold Dec 31 15:34:25.290895042 (624h5m25.161978077s) [seg_pos={0 0}, upload=546880]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:34:25.290990114 (624h5m25.162073159s) [upload=546880]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:34:25.290995724 (624h5m25.162078759s) [upload=546880]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:34:25.591032756 (624h5m25.462115791s) [upload=546880]: Done uploading segments... Dec 31 15:34:25.591049088 (624h5m25.462132123s) [upload=546880]: Upload finished. err: <nil> Duration: 30.00360016s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.............X..XX...........XXX..X.XX..XX..XXXXX...X.X.X...] duration: 3.501054215s trace: https://jaeger.dev.storj.io/trace/1901bc26af4982d6 FAIL (2025-12-31 15:34:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501080685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.X.........X..................] duration: 15.40588157s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:34:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 15.405913139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..............X..XX...........XXX..X.XX..XX..XXXXX...X.X.X..] duration: 3.734094328s trace: https://jaeger.dev.storj.io/trace/1901bc26af4982d6 FAIL (2025-12-31 15:33:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.734136929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..X] duration: 30.002676603s trace: https://jaeger.dev.storj.io/trace/5114a20f22515185 FAIL (2025-12-31 15:33:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4h4m14.270139612s) [upload=546860]: Uploading segments... Dec 31 15:33:14.399219637 (624h4m14.270302672s) [upload=546860]: Got next segment. Inline: false Dec 31 15:33:14.399324577 (624h4m14.270407612s) [upload=546860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:33:14.399329687 (624h4m14.270412722s) [upload=546860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:33:14.648560128 (624h4m14.519643163s) [seg_pos={0 0}, upload=546860]: Begin upload segment... Dec 31 15:33:14.648609011 (624h4m14.519692046s) [seg_pos={0 0}, upload=546860]: Done begin upload segment. Dec 31 15:33:14.648633337 (624h4m14.519716372s) [noise=true , address=145.79.227.4:4185 , piece_id=TLSXQCGS3WMAYOFQ5WKZH7MURIIIS7FML5ASVTSX4NTJD4DJWLDA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.648793682 (624h4m14.519876717s) [noise=true , address=45.91.116.5:4299 , piece_id=YYSK7OWMSKOE2L5WEOPGJMOZMYPT2I6ZSLKGMTZHQQ7YHEZN7A6Q, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.649417901 (624h4m14.520500936s) [noise=true , address=45.91.116.13:4156 , piece_id=JJJ5U4LR7ITHUOBC2B5KPZMRQS2C5YI2XOOGE2RG2YMKDV7WE7ZQ, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.649963049 (624h4m14.521046094s) [noise=true , address=45.91.116.14:4192 , piece_id=SGRVAPIIXL647X4EOE4LMUMD2GCJEK3V6IL7SYXECEHSDGELY7VA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.650299760 (624h4m14.521382795s) [noise=true , address=45.91.116.9:4262 , piece_id=PCDWCNOIWTISOLFPKOI4UD2JUNFNIQNCMEGGOFJWJMAW53XDK6DQ, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.650590233 (624h4m14.521673278s) [noise=true , address=145.79.227.41:4087 , piece_id=6D2NLV4MILQPXMLLEV2WFFWCMAN3VHSNU7ZXLCRDIIMTG6HMMHBA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.650876147 (624h4m14.521959193s) [noise=true , address=45.91.116.8:4045 , piece_id=DMQZOTFZ7XAX3IK6BZJLD7KA4RJIXYLBSY45PU6NZMGCBOOUG7UA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.651184425 (624h4m14.52226746s) [noise=true , address=145.79.227.45:4229 , piece_id=4HGQXJZIREXE44J226IB3OGSL2YXFN2CBEPP6F7LJDTTSCPRHZTA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.652323714 (624h4m14.523406749s) [noise=true , address=77.90.31.19:4844 , piece_id=KSQ24HCBW3THFZUM353I2OX6IC5NWZAKUI4CU5ZL7GQWSG5JJDWA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.652382846 (624h4m14.523465921s) [noise=true , address=45.91.116.6:4975 , piece_id=5QEKIYXRWJROXT5SX7KJRO5J625YOA3DFQ4ZN7CJNYEG4ALOYFLA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.653451231 (624h4m14.524534266s) [noise=true , address=145.79.227.3:4150 , piece_id=3AIIAO5TEMJQEJUSQFIGX32BYSOAZR7YA77AWOITI7DYHRSFASZQ, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.653736614 (624h4m14.524819659s) [noise=true , address=45.91.116.11:4120 , piece_id=QXO22D4FZ425QSC5ZR5LLBZBR5HZGBBPZLTW4ZJS6F2DIZYMNT5Q, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.654555975 (624h4m14.52563902s) [noise=true , address=45.91.116.12:4228 , piece_id=II7PZNL5JUX26LPKEW3IN27ACTOD5QIQQCDFUK6A24FHIDRAIQEQ, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.655594673 (624h4m14.526677718s) [noise=true , address=145.79.227.22:4510 , piece_id=Y76OXK6WTFFQF2X7MPX377MRXYUNYE3IK2NTYBSE3RDZZRHRRVGA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.656686091 (624h4m14.527769126s) [noise=true , address=77.90.31.17:4770 , piece_id=CTQWFT2GH2BIFN4SMDDEF3Q46BQ7QILWOYRG2W6RHPYBGGZ36ESA, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:14.657932514 (624h4m14.529015559s) [noise=true , address=145.79.227.48:4303 , piece_id=26T4WPFSITILDYSEFDVXECFKRC23UW4BV4NYT3HLW3ACGLK5FRQQ, seg_pos={0 0}, upload=546860]: Uploading piece... Dec 31 15:33:16.359756986 (624h4m16.230840021s) [upload=546860]: Next returned nil segment Dec 31 15:33:16.359775932 (624h4m16.230858967s) [upload=546860]: Waiting for error group managing segments... Dec 31 15:33:16.640750006 (624h4m16.511833041s) [noise=true , address=145.79.227.3:4150 , piece_id=3AIIAO5TEMJQEJUSQFIGX32BYSOAZR7YA77AWOITI7DYHRSFASZQ, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:16.642780913 (624h4m16.513863958s) [noise=true , address=145.79.227.22:4510 , piece_id=Y76OXK6WTFFQF2X7MPX377MRXYUNYE3IK2NTYBSE3RDZZRHRRVGA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:16.645734187 (624h4m16.516817222s) [noise=true , address=145.79.227.41:4087 , piece_id=6D2NLV4MILQPXMLLEV2WFFWCMAN3VHSNU7ZXLCRDIIMTG6HMMHBA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:16.649744693 (624h4m16.520827738s) [noise=true , address=145.79.227.45:4229 , piece_id=4HGQXJZIREXE44J226IB3OGSL2YXFN2CBEPP6F7LJDTTSCPRHZTA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:16.662502957 (624h4m16.533586002s) [noise=true , address=145.79.227.4:4185 , piece_id=TLSXQCGS3WMAYOFQ5WKZH7MURIIIS7FML5ASVTSX4NTJD4DJWLDA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:16.674205100 (624h4m16.545288135s) [noise=true , address=145.79.227.48:4303 , piece_id=26T4WPFSITILDYSEFDVXECFKRC23UW4BV4NYT3HLW3ACGLK5FRQQ, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:16.906907498 (624h4m16.777990543s) [noise=true , address=77.90.31.19:4844 , piece_id=KSQ24HCBW3THFZUM353I2OX6IC5NWZAKUI4CU5ZL7GQWSG5JJDWA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:16.967273654 (624h4m16.838356689s) [noise=true , address=45.91.116.11:4120 , piece_id=QXO22D4FZ425QSC5ZR5LLBZBR5HZGBBPZLTW4ZJS6F2DIZYMNT5Q, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:17.005630691 (624h4m16.876713736s) [noise=true , address=45.91.116.8:4045 , piece_id=DMQZOTFZ7XAX3IK6BZJLD7KA4RJIXYLBSY45PU6NZMGCBOOUG7UA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:17.043653031 (624h4m16.914736066s) [noise=true , address=45.91.116.9:4262 , piece_id=PCDWCNOIWTISOLFPKOI4UD2JUNFNIQNCMEGGOFJWJMAW53XDK6DQ, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:17.047212910 (624h4m16.918295945s) [noise=true , address=45.91.116.13:4156 , piece_id=JJJ5U4LR7ITHUOBC2B5KPZMRQS2C5YI2XOOGE2RG2YMKDV7WE7ZQ, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:17.051047932 (624h4m16.922130977s) [noise=true , address=45.91.116.12:4228 , piece_id=II7PZNL5JUX26LPKEW3IN27ACTOD5QIQQCDFUK6A24FHIDRAIQEQ, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:17.268825611 (624h4m17.139908646s) [noise=true , address=77.90.31.17:4770 , piece_id=CTQWFT2GH2BIFN4SMDDEF3Q46BQ7QILWOYRG2W6RHPYBGGZ36ESA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: <nil> Dec 31 15:33:17.268847212 (624h4m17.139930257s) [seg_pos={0 0}, upload=546860]: Segment reached optimal threshold of 13 pieces. Dec 31 15:33:17.269142134 (624h4m17.140225179s) [noise=true , address=45.91.116.14:4192 , piece_id=SGRVAPIIXL647X4EOE4LMUMD2GCJEK3V6IL7SYXECEHSDGELY7VA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 31 15:33:17.269150730 (624h4m17.140233775s) [noise=true , address=45.91.116.14:4192 , piece_id=SGRVAPIIXL647X4EOE4LMUMD2GCJEK3V6IL7SYXECEHSDGELY7VA, seg_pos={0 0}, upload=546860]: Long tail cancellation due to optimal threshold Dec 31 15:33:17.269312178 (624h4m17.140395223s) [noise=true , address=45.91.116.6:4975 , piece_id=5QEKIYXRWJROXT5SX7KJRO5J625YOA3DFQ4ZN7CJNYEG4ALOYFLA, seg_pos={0 0}, upload=546860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:58198->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:58198->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58198->45.91.116.6:4975: use of closed network connection Dec 31 15:33:17.269321315 (624h4m17.14040436s) [noise=true , address=45.91.116.6:4975 , piece_id=5QEKIYXRWJROXT5SX7KJRO5J625YOA3DFQ4ZN7CJNYEG4ALOYFLA, seg_pos={0 0}, upload=546860]: Long tail cancellation due to optimal threshold Dec 31 15:33:17.269350450 (624h4m17.140433486s) [noise=true , address=45.91.116.5:4299 , piece_id=YYSK7OWMSKOE2L5WEOPGJMOZMYPT2I6ZSLKGMTZHQQ7YHEZN7A6Q, seg_pos={0 0}, upload=546860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:33260->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:33260->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33260->45.91.116.5:4299: use of closed network connection Dec 31 15:33:17.269380126 (624h4m17.140463161s) [noise=true , address=45.91.116.5:4299 , piece_id=YYSK7OWMSKOE2L5WEOPGJMOZMYPT2I6ZSLKGMTZHQQ7YHEZN7A6Q, seg_pos={0 0}, upload=546860]: Long tail cancellation due to optimal threshold Dec 31 15:33:17.269403902 (624h4m17.140486947s) [seg_pos={0 0}, upload=546860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:33:17.269511727 (624h4m17.140594762s) [upload=546860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:33:17.269517007 (624h4m17.140600052s) [upload=546860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:33:17.565308386 (624h4m17.436391431s) [upload=546860]: Done uploading segments... Dec 31 15:33:17.565336259 (624h4m17.436419304s) [upload=546860]: Upload finished. err: <nil> Duration: 30.002697322s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...............X..XX...........XXX..X.XX..XX..XXXXX...X.X.X.] duration: 3.724234016s trace: https://jaeger.dev.storj.io/trace/1901bc26af4982d6 FAIL (2025-12-31 15:33:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.72425815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.X.........X.................] duration: 29.496083494s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:33:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 29.496098474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X..XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX..] duration: 28.516036105s trace: https://jaeger.dev.storj.io/trace/4937d4a534794678 FAIL (2025-12-31 15:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ) context deadline exceeded ======LOGS====== Dec 31 15:30:36.536837632 (624h1m36.407920667s) [upload=546822]: Starting upload Dec 31 15:30:36.536889531 (624h1m36.407972566s) [upload=546822]: Done starting upload Dec 31 15:30:36.537270126 (624h1m36.408353161s) [upload=546822]: Uploading segments... Dec 31 15:30:36.537776961 (624h1m36.408859996s) [upload=546822]: Got next segment. Inline: false Dec 31 15:30:36.537866051 (624h1m36.408949086s) [upload=546822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:30:36.537873785 (624h1m36.40895684s) [upload=546822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:30:36.787554543 (624h1m36.658637578s) [seg_pos={0 0}, upload=546822]: Begin upload segment... Dec 31 15:30:36.787603396 (624h1m36.658686441s) [seg_pos={0 0}, upload=546822]: Done begin upload segment. Dec 31 15:30:36.787651207 (624h1m36.658734242s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.787700390 (624h1m36.658783426s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.788793662 (624h1m36.659876697s) [noise=true , address=145.79.227.50:4375 , piece_id=7SF7XJO7VDMISLX7KYQEURY2YFJ46BRH536WR67KCYSJB4VYFMZA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.788873655 (624h1m36.65995669s) [noise=true , address=145.79.227.14:4837 , piece_id=5VODDVFVISLVFQCEUXQYA55RIUR6EXTQECKGR7EZ5EXKFIYWSGTQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789441426 (624h1m36.660524461s) [noise=true , address=77.90.31.19:4842 , piece_id=2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789524113 (624h1m36.660607148s) [noise=true , address=145.79.227.27:4691 , piece_id=7TFLNQ33JVNVBY4H5UVBWVUS7MVHGGBHSCZBT2CBSMJ7AXRAL2UQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789851387 (624h1m36.660934462s) [noise=true , address=45.91.116.9:4263 , piece_id=3XGLNRPPOAX7MR75JPOKPJZFI35LSB45ZGFHAHS7RUJDCMXZD5LQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790206022 (624h1m36.661289057s) [noise=true , address=77.90.31.13:4627 , piece_id=HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790238964 (624h1m36.66132206s) [noise=true , address=145.79.227.45:4231 , piece_id=DTJN4TQB4WRSZ3RBCYQWJKX25ESM52QBIIYP3EGDERRPG2WKMHUQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790509029 (624h1m36.661592074s) [noise=true , address=145.79.227.6:4257 , piece_id=ZLAXNUQIGOPEZYZV6VYIJO53RK6JAM5A2QSVSVHUNVMM6TIWYI3A, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790886657 (624h1m36.661969692s) [noise=true , address=77.90.31.10:4519 , piece_id=3XAV23HJBJFQFYOKUKWUYZ6N55A52D7GT5JTVSJPKARSOACGPDBA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790904652 (624h1m36.661987697s) [noise=true , address=45.91.116.10:4081 , piece_id=DBH3HAZQDTDVOC6ZNLOVGEDCNSVZJZDRRIPWSK4GJETDRLSMBXFQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791228047 (624h1m36.662311092s) [noise=true , address=77.90.31.11:4556 , piece_id=QU232AMXPG5B5C677VBKSQKUBJNLLFPUSCEYFFWTOYVRPVWLN7NQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791532678 (624h1m36.662615713s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791554981 (624h1m36.662638026s) [noise=true , address=77.90.31.21:4880 , piece_id=XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.792290140 (624h1m36.663373185s) [noise=true , address=45.91.116.5:4299 , piece_id=DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:38.494768429 (624h1m38.365851474s) [upload=546822]: Next returned nil segment Dec 31 15:30:38.494783277 (624h1m38.365866322s) [upload=546822]: Waiting for error group managing segments... Dec 31 15:30:38.770858203 (624h1m38.641941248s) [noise=true , address=145.79.227.14:4837 , piece_id=5VODDVFVISLVFQCEUXQYA55RIUR6EXTQECKGR7EZ5EXKFIYWSGTQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.778059095 (624h1m38.64914213s) [noise=true , address=145.79.227.27:4691 , piece_id=7TFLNQ33JVNVBY4H5UVBWVUS7MVHGGBHSCZBT2CBSMJ7AXRAL2UQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.779089736 (624h1m38.650172782s) [noise=true , address=145.79.227.50:4375 , piece_id=7SF7XJO7VDMISLX7KYQEURY2YFJ46BRH536WR67KCYSJB4VYFMZA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.781625004 (624h1m38.652708039s) [noise=true , address=145.79.227.6:4257 , piece_id=ZLAXNUQIGOPEZYZV6VYIJO53RK6JAM5A2QSVSVHUNVMM6TIWYI3A, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.782076324 (624h1m38.653159359s) [noise=true , address=145.79.227.45:4231 , piece_id=DTJN4TQB4WRSZ3RBCYQWJKX25ESM52QBIIYP3EGDERRPG2WKMHUQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.963382376 (624h1m38.834465411s) [noise=true , address=77.90.31.13:4627 , piece_id=HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.988745201 (624h1m38.859828236s) [noise=true , address=77.90.31.21:4880 , piece_id=XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.042592644 (624h1m38.913675689s) [noise=true , address=77.90.31.11:4556 , piece_id=QU232AMXPG5B5C677VBKSQKUBJNLLFPUSCEYFFWTOYVRPVWLN7NQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.057757219 (624h1m38.928840264s) [noise=true , address=77.90.31.10:4519 , piece_id=3XAV23HJBJFQFYOKUKWUYZ6N55A52D7GT5JTVSJPKARSOACGPDBA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.102567915 (624h1m38.97365096s) [noise=true , address=45.91.116.9:4263 , piece_id=3XGLNRPPOAX7MR75JPOKPJZFI35LSB45ZGFHAHS7RUJDCMXZD5LQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.194835856 (624h1m39.065918891s) [noise=true , address=45.91.116.5:4299 , piece_id=DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.197351036 (624h1m39.068434071s) [noise=true , address=45.91.116.10:4081 , piece_id=DBH3HAZQDTDVOC6ZNLOVGEDCNSVZJZDRRIPWSK4GJETDRLSMBXFQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.217897980 (624h1m39.088981015s) [noise=true , address=77.90.31.19:4842 , piece_id=2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.217923208 (624h1m39.089006243s) [seg_pos={0 0}, upload=546822]: Segment reached optimal threshold of 13 pieces. Dec 31 15:30:39.218084836 (624h1m39.089167881s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.218090477 (624h1m39.089173512s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222078690 (624h1m39.093161725s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.222093669 (624h1m39.093176704s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222223726 (624h1m39.093306771s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.222229617 (624h1m39.093312652s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222245388 (624h1m39.093328433s) [seg_pos={0 0}, upload=546822]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:30:39.222326632 (624h1m39.093409667s) [upload=546822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:30:39.222333004 (624h1m39.093416039s) [upload=546822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:30:39.518262636 (624h1m39.389345671s) [upload=546822]: Done uploading segments... Dec 31 15:30:39.518282044 (624h1m39.389365079s) [upload=546822]: Upload finished. err: <nil> Duration: 28.516061804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [................X..XX...........XXX..X.XX..XX..XXXXX...X.X.X] duration: 5.002449492s trace: https://jaeger.dev.storj.io/trace/1901bc26af4982d6 FAIL (2025-12-31 15:33:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002473979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.X.........X................] duration: 41.237828989s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:32:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 41.237954038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.................X..XX...........XXX..X.XX..XX..XXXXX...X.X.] duration: 4.772115072s trace: https://jaeger.dev.storj.io/trace/2865769b559d197e FAIL (2025-12-31 15:32:19 UTC) orbiter warning: context deadline exceeded Duration: 4.772143174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.X.........X...............] duration: 35.357566108s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 35.35759285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X.................X..XX...........XXX..X.XX..XX..XXXXX...X.X] duration: 5.001542823s trace: https://jaeger.dev.storj.io/trace/2865769b559d197e FAIL (2025-12-31 15:32:04 UTC) orbiter warning: context deadline exceeded Duration: 5.0015733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X..XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX.] duration: 5.955739077s trace: https://jaeger.dev.storj.io/trace/4937d4a534794678 FAIL (2025-12-31 15:31:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ) context deadline exceeded ======LOGS====== Dec 31 15:30:36.536837632 (624h1m36.407920667s) [upload=546822]: Starting upload Dec 31 15:30:36.536889531 (624h1m36.407972566s) [upload=546822]: Done starting upload Dec 31 15:30:36.537270126 (624h1m36.408353161s) [upload=546822]: Uploading segments... Dec 31 15:30:36.537776961 (624h1m36.408859996s) [upload=546822]: Got next segment. Inline: false Dec 31 15:30:36.537866051 (624h1m36.408949086s) [upload=546822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:30:36.537873785 (624h1m36.40895684s) [upload=546822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:30:36.787554543 (624h1m36.658637578s) [seg_pos={0 0}, upload=546822]: Begin upload segment... Dec 31 15:30:36.787603396 (624h1m36.658686441s) [seg_pos={0 0}, upload=546822]: Done begin upload segment. Dec 31 15:30:36.787651207 (624h1m36.658734242s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.787700390 (624h1m36.658783426s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.788793662 (624h1m36.659876697s) [noise=true , address=145.79.227.50:4375 , piece_id=7SF7XJO7VDMISLX7KYQEURY2YFJ46BRH536WR67KCYSJB4VYFMZA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.788873655 (624h1m36.65995669s) [noise=true , address=145.79.227.14:4837 , piece_id=5VODDVFVISLVFQCEUXQYA55RIUR6EXTQECKGR7EZ5EXKFIYWSGTQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789441426 (624h1m36.660524461s) [noise=true , address=77.90.31.19:4842 , piece_id=2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789524113 (624h1m36.660607148s) [noise=true , address=145.79.227.27:4691 , piece_id=7TFLNQ33JVNVBY4H5UVBWVUS7MVHGGBHSCZBT2CBSMJ7AXRAL2UQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789851387 (624h1m36.660934462s) [noise=true , address=45.91.116.9:4263 , piece_id=3XGLNRPPOAX7MR75JPOKPJZFI35LSB45ZGFHAHS7RUJDCMXZD5LQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790206022 (624h1m36.661289057s) [noise=true , address=77.90.31.13:4627 , piece_id=HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790238964 (624h1m36.66132206s) [noise=true , address=145.79.227.45:4231 , piece_id=DTJN4TQB4WRSZ3RBCYQWJKX25ESM52QBIIYP3EGDERRPG2WKMHUQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790509029 (624h1m36.661592074s) [noise=true , address=145.79.227.6:4257 , piece_id=ZLAXNUQIGOPEZYZV6VYIJO53RK6JAM5A2QSVSVHUNVMM6TIWYI3A, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790886657 (624h1m36.661969692s) [noise=true , address=77.90.31.10:4519 , piece_id=3XAV23HJBJFQFYOKUKWUYZ6N55A52D7GT5JTVSJPKARSOACGPDBA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790904652 (624h1m36.661987697s) [noise=true , address=45.91.116.10:4081 , piece_id=DBH3HAZQDTDVOC6ZNLOVGEDCNSVZJZDRRIPWSK4GJETDRLSMBXFQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791228047 (624h1m36.662311092s) [noise=true , address=77.90.31.11:4556 , piece_id=QU232AMXPG5B5C677VBKSQKUBJNLLFPUSCEYFFWTOYVRPVWLN7NQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791532678 (624h1m36.662615713s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791554981 (624h1m36.662638026s) [noise=true , address=77.90.31.21:4880 , piece_id=XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.792290140 (624h1m36.663373185s) [noise=true , address=45.91.116.5:4299 , piece_id=DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:38.494768429 (624h1m38.365851474s) [upload=546822]: Next returned nil segment Dec 31 15:30:38.494783277 (624h1m38.365866322s) [upload=546822]: Waiting for error group managing segments... Dec 31 15:30:38.770858203 (624h1m38.641941248s) [noise=true , address=145.79.227.14:4837 , piece_id=5VODDVFVISLVFQCEUXQYA55RIUR6EXTQECKGR7EZ5EXKFIYWSGTQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.778059095 (624h1m38.64914213s) [noise=true , address=145.79.227.27:4691 , piece_id=7TFLNQ33JVNVBY4H5UVBWVUS7MVHGGBHSCZBT2CBSMJ7AXRAL2UQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.779089736 (624h1m38.650172782s) [noise=true , address=145.79.227.50:4375 , piece_id=7SF7XJO7VDMISLX7KYQEURY2YFJ46BRH536WR67KCYSJB4VYFMZA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.781625004 (624h1m38.652708039s) [noise=true , address=145.79.227.6:4257 , piece_id=ZLAXNUQIGOPEZYZV6VYIJO53RK6JAM5A2QSVSVHUNVMM6TIWYI3A, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.782076324 (624h1m38.653159359s) [noise=true , address=145.79.227.45:4231 , piece_id=DTJN4TQB4WRSZ3RBCYQWJKX25ESM52QBIIYP3EGDERRPG2WKMHUQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.963382376 (624h1m38.834465411s) [noise=true , address=77.90.31.13:4627 , piece_id=HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.988745201 (624h1m38.859828236s) [noise=true , address=77.90.31.21:4880 , piece_id=XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.042592644 (624h1m38.913675689s) [noise=true , address=77.90.31.11:4556 , piece_id=QU232AMXPG5B5C677VBKSQKUBJNLLFPUSCEYFFWTOYVRPVWLN7NQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.057757219 (624h1m38.928840264s) [noise=true , address=77.90.31.10:4519 , piece_id=3XAV23HJBJFQFYOKUKWUYZ6N55A52D7GT5JTVSJPKARSOACGPDBA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.102567915 (624h1m38.97365096s) [noise=true , address=45.91.116.9:4263 , piece_id=3XGLNRPPOAX7MR75JPOKPJZFI35LSB45ZGFHAHS7RUJDCMXZD5LQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.194835856 (624h1m39.065918891s) [noise=true , address=45.91.116.5:4299 , piece_id=DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.197351036 (624h1m39.068434071s) [noise=true , address=45.91.116.10:4081 , piece_id=DBH3HAZQDTDVOC6ZNLOVGEDCNSVZJZDRRIPWSK4GJETDRLSMBXFQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.217897980 (624h1m39.088981015s) [noise=true , address=77.90.31.19:4842 , piece_id=2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.217923208 (624h1m39.089006243s) [seg_pos={0 0}, upload=546822]: Segment reached optimal threshold of 13 pieces. Dec 31 15:30:39.218084836 (624h1m39.089167881s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.218090477 (624h1m39.089173512s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222078690 (624h1m39.093161725s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.222093669 (624h1m39.093176704s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222223726 (624h1m39.093306771s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.222229617 (624h1m39.093312652s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222245388 (624h1m39.093328433s) [seg_pos={0 0}, upload=546822]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:30:39.222326632 (624h1m39.093409667s) [upload=546822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:30:39.222333004 (624h1m39.093416039s) [upload=546822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:30:39.518262636 (624h1m39.389345671s) [upload=546822]: Done uploading segments... Dec 31 15:30:39.518282044 (624h1m39.389365079s) [upload=546822]: Upload finished. err: <nil> Duration: 5.955767642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XX.................X..XX...........XXX..X.XX..XX..XXXXX...X.] duration: 3.819566625s trace: https://jaeger.dev.storj.io/trace/63fe59803d758290 FAIL (2025-12-31 15:31:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.819596372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..X..XXX......XX......X..X.XX.XX.XX..X..X..X..XX.X.....XX] duration: 30.003151062s trace: https://jaeger.dev.storj.io/trace/4937d4a534794678 FAIL (2025-12-31 15:31:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ) context deadline exceeded ======LOGS====== Dec 31 15:30:36.536837632 (624h1m36.407920667s) [upload=546822]: Starting upload Dec 31 15:30:36.536889531 (624h1m36.407972566s) [upload=546822]: Done starting upload Dec 31 15:30:36.537270126 (624h1m36.408353161s) [upload=546822]: Uploading segments... Dec 31 15:30:36.537776961 (624h1m36.408859996s) [upload=546822]: Got next segment. Inline: false Dec 31 15:30:36.537866051 (624h1m36.408949086s) [upload=546822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 15:30:36.537873785 (624h1m36.40895684s) [upload=546822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 15:30:36.787554543 (624h1m36.658637578s) [seg_pos={0 0}, upload=546822]: Begin upload segment... Dec 31 15:30:36.787603396 (624h1m36.658686441s) [seg_pos={0 0}, upload=546822]: Done begin upload segment. Dec 31 15:30:36.787651207 (624h1m36.658734242s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.787700390 (624h1m36.658783426s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.788793662 (624h1m36.659876697s) [noise=true , address=145.79.227.50:4375 , piece_id=7SF7XJO7VDMISLX7KYQEURY2YFJ46BRH536WR67KCYSJB4VYFMZA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.788873655 (624h1m36.65995669s) [noise=true , address=145.79.227.14:4837 , piece_id=5VODDVFVISLVFQCEUXQYA55RIUR6EXTQECKGR7EZ5EXKFIYWSGTQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789441426 (624h1m36.660524461s) [noise=true , address=77.90.31.19:4842 , piece_id=2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789524113 (624h1m36.660607148s) [noise=true , address=145.79.227.27:4691 , piece_id=7TFLNQ33JVNVBY4H5UVBWVUS7MVHGGBHSCZBT2CBSMJ7AXRAL2UQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.789851387 (624h1m36.660934462s) [noise=true , address=45.91.116.9:4263 , piece_id=3XGLNRPPOAX7MR75JPOKPJZFI35LSB45ZGFHAHS7RUJDCMXZD5LQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790206022 (624h1m36.661289057s) [noise=true , address=77.90.31.13:4627 , piece_id=HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790238964 (624h1m36.66132206s) [noise=true , address=145.79.227.45:4231 , piece_id=DTJN4TQB4WRSZ3RBCYQWJKX25ESM52QBIIYP3EGDERRPG2WKMHUQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790509029 (624h1m36.661592074s) [noise=true , address=145.79.227.6:4257 , piece_id=ZLAXNUQIGOPEZYZV6VYIJO53RK6JAM5A2QSVSVHUNVMM6TIWYI3A, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790886657 (624h1m36.661969692s) [noise=true , address=77.90.31.10:4519 , piece_id=3XAV23HJBJFQFYOKUKWUYZ6N55A52D7GT5JTVSJPKARSOACGPDBA, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.790904652 (624h1m36.661987697s) [noise=true , address=45.91.116.10:4081 , piece_id=DBH3HAZQDTDVOC6ZNLOVGEDCNSVZJZDRRIPWSK4GJETDRLSMBXFQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791228047 (624h1m36.662311092s) [noise=true , address=77.90.31.11:4556 , piece_id=QU232AMXPG5B5C677VBKSQKUBJNLLFPUSCEYFFWTOYVRPVWLN7NQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791532678 (624h1m36.662615713s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.791554981 (624h1m36.662638026s) [noise=true , address=77.90.31.21:4880 , piece_id=XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:36.792290140 (624h1m36.663373185s) [noise=true , address=45.91.116.5:4299 , piece_id=DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A, seg_pos={0 0}, upload=546822]: Uploading piece... Dec 31 15:30:38.494768429 (624h1m38.365851474s) [upload=546822]: Next returned nil segment Dec 31 15:30:38.494783277 (624h1m38.365866322s) [upload=546822]: Waiting for error group managing segments... Dec 31 15:30:38.770858203 (624h1m38.641941248s) [noise=true , address=145.79.227.14:4837 , piece_id=5VODDVFVISLVFQCEUXQYA55RIUR6EXTQECKGR7EZ5EXKFIYWSGTQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.778059095 (624h1m38.64914213s) [noise=true , address=145.79.227.27:4691 , piece_id=7TFLNQ33JVNVBY4H5UVBWVUS7MVHGGBHSCZBT2CBSMJ7AXRAL2UQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.779089736 (624h1m38.650172782s) [noise=true , address=145.79.227.50:4375 , piece_id=7SF7XJO7VDMISLX7KYQEURY2YFJ46BRH536WR67KCYSJB4VYFMZA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.781625004 (624h1m38.652708039s) [noise=true , address=145.79.227.6:4257 , piece_id=ZLAXNUQIGOPEZYZV6VYIJO53RK6JAM5A2QSVSVHUNVMM6TIWYI3A, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.782076324 (624h1m38.653159359s) [noise=true , address=145.79.227.45:4231 , piece_id=DTJN4TQB4WRSZ3RBCYQWJKX25ESM52QBIIYP3EGDERRPG2WKMHUQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.963382376 (624h1m38.834465411s) [noise=true , address=77.90.31.13:4627 , piece_id=HY3HRS7UD65QEFOYMCPWGBNAJ3VTE5XYXWEEXVR5GG4YW2FCRROA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:38.988745201 (624h1m38.859828236s) [noise=true , address=77.90.31.21:4880 , piece_id=XL2HGUFDK6ORYOJQ4VJPKDGZNE4JILOVHDNZ32XI37YE3GZZXMFQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.042592644 (624h1m38.913675689s) [noise=true , address=77.90.31.11:4556 , piece_id=QU232AMXPG5B5C677VBKSQKUBJNLLFPUSCEYFFWTOYVRPVWLN7NQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.057757219 (624h1m38.928840264s) [noise=true , address=77.90.31.10:4519 , piece_id=3XAV23HJBJFQFYOKUKWUYZ6N55A52D7GT5JTVSJPKARSOACGPDBA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.102567915 (624h1m38.97365096s) [noise=true , address=45.91.116.9:4263 , piece_id=3XGLNRPPOAX7MR75JPOKPJZFI35LSB45ZGFHAHS7RUJDCMXZD5LQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.194835856 (624h1m39.065918891s) [noise=true , address=45.91.116.5:4299 , piece_id=DOZY6ETFFA3UNUPGK3UV3G3LCTEWW2QYDDUN4OAMS6P5TMAMJ32A, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.197351036 (624h1m39.068434071s) [noise=true , address=45.91.116.10:4081 , piece_id=DBH3HAZQDTDVOC6ZNLOVGEDCNSVZJZDRRIPWSK4GJETDRLSMBXFQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.217897980 (624h1m39.088981015s) [noise=true , address=77.90.31.19:4842 , piece_id=2U3SWQNXMJ54CFG4A42T3E4IAA3W4PB5BVQUN265HOW6BGH6JVGQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: <nil> Dec 31 15:30:39.217923208 (624h1m39.089006243s) [seg_pos={0 0}, upload=546822]: Segment reached optimal threshold of 13 pieces. Dec 31 15:30:39.218084836 (624h1m39.089167881s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.218090477 (624h1m39.089173512s) [noise=true , address=77.90.31.15:4700 , piece_id=HMLUCF6F4V7KMWED3MKIAPEFT5UCSUPQG2JY2NWDPDFDFCJHZVUA, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222078690 (624h1m39.093161725s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.222093669 (624h1m39.093176704s) [noise=true , address=145.79.227.13:4803 , piece_id=MGVPC4CJWOGWB54YOBGX6ISKTVTPC5ZV47SOHARF3NXY3MT7TF5Q, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222223726 (624h1m39.093306771s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 31 15:30:39.222229617 (624h1m39.093312652s) [noise=true , address=77.90.31.14:4663 , piece_id=UJ23VJZWOKLKXMW66KLLMY7AI7XDDZLOLXKQ2FMAJJL4I2H4OUYQ, seg_pos={0 0}, upload=546822]: Long tail cancellation due to optimal threshold Dec 31 15:30:39.222245388 (624h1m39.093328433s) [seg_pos={0 0}, upload=546822]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 15:30:39.222326632 (624h1m39.093409667s) [upload=546822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:30:39.222333004 (624h1m39.093416039s) [upload=546822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:30:39.518262636 (624h1m39.389345671s) [upload=546822]: Done uploading segments... Dec 31 15:30:39.518282044 (624h1m39.389365079s) [upload=546822]: Upload finished. err: <nil> Duration: 30.003176228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.X.........X..............] duration: 34.834596114s trace: https://jaeger.dev.storj.io/trace/70af76874d72c3d6 FAIL (2025-12-31 15:31:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044531632 (623h49m39.915614677s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.044887860 (623h49m39.915970895s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045315133 (623h49m39.916398168s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.045906780 (623h49m39.916989825s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046286783 (623h49m39.917369828s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.046683379 (623h49m39.917766424s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047051570 (623h49m39.918134605s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047411756 (623h49m39.918494801s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.047832678 (623h49m39.918915723s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.048299296 (623h49m39.919382341s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.050007608 (623h49m39.921090674s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.051392496 (623h49m39.922475531s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.055661204 (623h49m39.92674425s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.057383443 (623h49m39.928466489s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:40.061501685 (623h49m39.93258473s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Uploading piece... Dec 31 15:18:41.462046371 (623h49m41.333129416s) [upload=546645]: Next returned nil segment Dec 31 15:18:41.462062342 (623h49m41.333145387s) [upload=546645]: Waiting for error group managing segments... Dec 31 15:18:41.821355907 (623h49m41.692438952s) [noise=true , address=64.49.13.27:4962 , piece_id=X4AE5NV2GHKSZZPFM4JWH4ZDRRAYFMSS27SUZ5VZI3VOCCRE2S2A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:41.931431837 (623h49m41.802514882s) [noise=true , address=145.79.227.19:4086 , piece_id=RNGZNHKOR5PON2MLKLRHQH6E7UF6A5MSVNYQEK44OVUFXAXISWUA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.103444453 (623h49m41.974527488s) [noise=true , address=77.90.31.12:4619 , piece_id=XIW4AS5VROVYYQ76I52CV2SEFAMS4RL4KZUQXKVM3ONIP5UJ3LPA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.163984480 (623h49m42.035067515s) [noise=true , address=77.90.31.17:4780 , piece_id=S3QXIMI76WPOXFP3BKQRQW2ITXG4EO53EVDO2IREDNZK2KTGMRDA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.185245334 (623h49m42.056328369s) [noise=true , address=145.79.227.12:4779 , piece_id=HLK3UGW7URXSJRGA33A43SZQVYXHQ7DCI25UHWRN6WRTEACXU2EQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.213185806 (623h49m42.084268851s) [noise=true , address=77.90.31.18:4813 , piece_id=EK6FOZCWB54FBNVVEEORJJVPTLNH5YY2DR3UZV7BI27X6DAHOKPQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.280525792 (623h49m42.151608827s) [noise=true , address=45.91.116.7:4033 , piece_id=N34AZLFWATUW4XC37NVGLNK2A5NSYXU7S3KUWUWKRF3A7HE6IKMQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.320822542 (623h49m42.191905577s) [noise=true , address=77.90.31.13:4657 , piece_id=Z35CBAIFXASWTD27L5M2BSH2R3Q2EJEVILHO3PFLZUNUW5MQLB7A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.774697664 (623h49m42.645780699s) [noise=true , address=145.79.227.43:4169 , piece_id=SKRYGDV4FK66IC7QRBT2DOXYABMFUJADVSHNHJT56CA6CFT6SD7Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.860347003 (623h49m42.731430038s) [noise=true , address=145.79.227.35:4515 , piece_id=2AT7TLS76XSHHENPGIYG3CEFEEWXOHBP3CH5NDSRLGYONBUNFD4A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.866157006 (623h49m42.737240051s) [noise=true , address=145.79.227.33:4883 , piece_id=5EJ2WSDUFYQYKLE6BX3TJ2DARMPG5W3JT2VCJG7KYI5RBPI5ECRA, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.932410953 (623h49m42.803493988s) [noise=true , address=45.91.116.4:4967 , piece_id=4GBXFLB2JGRWU7W3EJPZWXEHHAX2NXKZDDS4T2MPKOEEGG7C4F2Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.935986601 (623h49m42.807069646s) [noise=true , address=145.79.227.49:4346 , piece_id=YDPVURS7RYU647ZUPID5Q2KMGE347ETBI54FJONUT35V6MIY6QEQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941089999 (623h49m42.812173044s) [noise=true , address=145.79.227.44:4219 , piece_id=7RMKNMBB5US57KCRVTAEDVBO6D3UIIZVZ34SYGUSYM6DYTCRBVAQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: <nil> Dec 31 15:18:42.941112031 (623h49m42.812195077s) [seg_pos={0 0}, upload=546645]: Segment reached optimal threshold of 14 pieces. Dec 31 15:18:42.941299107 (623h49m42.812382152s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45766->64.49.13.169:4706: use of closed network connection Dec 31 15:18:42.941305269 (623h49m42.812388304s) [noise=true , address=64.49.13.169:4706 , piece_id=AISTBQRIXUKBREU3LWDUYVCPFGMXLOGS7ABDFATFFZOESOF2Z67Q, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941498858 (623h49m42.812581903s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941507013 (623h49m42.812590058s) [noise=true , address=145.79.227.30:4773 , piece_id=OL4TDR3MRN5NQ7TAF2NA5MDCODXANKO4OFRQXEWSMEXJD4HAJ7AQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941757490 (623h49m42.812840535s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941829818 (623h49m42.812912863s) [noise=true , address=45.91.116.12:4250 , piece_id=HNTYFGP2DLWSROSEQPYT34HIJ337BHLILPULMMJNC2IKKQ2SJGWQ, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941844376 (623h49m42.812927411s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou): protocol: expected piece hash; context canceled; EOF Dec 31 15:18:42.941852321 (623h49m42.812935356s) [noise=true , address=145.79.227.34:4933 , piece_id=CHTZ6ZVWIVME5OH267D35YWKTGF2LJMV5C3WKPBGJIKKRLY2DA6A, seg_pos={0 0}, upload=546645]: Long tail cancellation due to optimal threshold Dec 31 15:18:42.941868972 (623h49m42.812952007s) [seg_pos={0 0}, upload=546645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 15:18:42.941946731 (623h49m42.813029766s) [upload=546645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 15:18:42.941952031 (623h49m42.813035076s) [upload=546645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 15:18:43.235127688 (623h49m43.106210723s) [upload=546645]: Done uploading segments... Dec 31 15:18:43.235146994 (623h49m43.106230029s) [upload=546645]: Upload finished. err: <nil> Duration: 34.834617035s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)