Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X............X.....X.X............X.........X.....X.....X.] duration: 3.489394093s trace: https://jaeger.dev.storj.io/trace/574930f7cdbe9bfb FAIL (2025-12-25 16:31:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.489421757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X............X.....X.X............X.........X.....X.....X] duration: 5.001304912s trace: https://jaeger.dev.storj.io/trace/574930f7cdbe9bfb FAIL (2025-12-25 16:31:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001326542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.....X....................................] duration: 5.799928685s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:30:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.799960977s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X............X.....X.X............X.........X.....X.....] duration: 3.753003633s trace: https://jaeger.dev.storj.io/trace/219373bdceb6a981 FAIL (2025-12-25 16:30:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.753105036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.....X...................................] duration: 5.792915535s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:30:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.792943076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.....X..................................] duration: 5.779026563s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:30:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.779054736s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X............X.....X.X............X.........X.....X....] duration: 4.130541677s trace: https://jaeger.dev.storj.io/trace/219373bdceb6a981 FAIL (2025-12-25 16:29:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.130572834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...................X.....X.................................] duration: 5.794621468s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:28:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.7946594s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X............X.....X.X............X.........X.....X...] duration: 3.392493769s trace: https://jaeger.dev.storj.io/trace/219373bdceb6a981 FAIL (2025-12-25 16:28:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.392526901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...................X.....X................................] duration: 5.775272595s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:28:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.775297292s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......X............X.....X.X............X.........X.....X..] duration: 3.420251584s trace: https://jaeger.dev.storj.io/trace/219373bdceb6a981 FAIL (2025-12-25 16:27:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42026482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...................X.....X...............................] duration: 5.72195037s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:27:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.721980138s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........X............X.....X.X............X.........X.....X.] duration: 5.000713766s trace: https://jaeger.dev.storj.io/trace/219373bdceb6a981 FAIL (2025-12-25 16:26:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000774751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...................X.....X..............................] duration: 5.82041932s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:26:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.820444708s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X............X.....X.X............X.........X.....X] duration: 5.001169093s trace: https://jaeger.dev.storj.io/trace/219373bdceb6a981 FAIL (2025-12-25 16:25:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001191325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...................X.....X.............................] duration: 5.922746051s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:25:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.922776419s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X............X.....X.X............X.........X.....] duration: 3.557186634s trace: https://jaeger.dev.storj.io/trace/6a401d590f788f0f FAIL (2025-12-25 16:25:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.557213245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........X............X.....X.X............X.........X....] duration: 5.001338636s trace: https://jaeger.dev.storj.io/trace/6a401d590f788f0f FAIL (2025-12-25 16:24:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001357993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...................X.....X............................] duration: 5.762940723s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:24:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.762986962s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X............X.....X.X............X.........X...] duration: 3.599238422s trace: https://jaeger.dev.storj.io/trace/6a401d590f788f0f FAIL (2025-12-25 16:23:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.599261096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...................X.....X...........................] duration: 5.781199971s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:23:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.781229237s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X............X.....X.X............X.........X..] duration: 4.881114188s trace: https://jaeger.dev.storj.io/trace/6a401d590f788f0f FAIL (2025-12-25 16:22:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.881135238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...................X.....X..........................] duration: 5.779328366s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:22:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.779361108s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............X............X.....X.X............X.........X.] duration: 3.49593731s trace: https://jaeger.dev.storj.io/trace/6a401d590f788f0f FAIL (2025-12-25 16:21:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495962759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...................X.....X.........................] duration: 5.756173747s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:21:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.756198735s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............X............X.....X.X............X.........X] duration: 5.001498278s trace: https://jaeger.dev.storj.io/trace/6a401d590f788f0f FAIL (2025-12-25 16:20:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001523166s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X............X.....X.X............X.........] duration: 3.410719109s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:20:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.410740348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...................X.....X........................] duration: 5.695947086s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:20:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.696039971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X...................X.....X.......................] duration: 5.775784598s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.775817602s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................X............X.....X.X............X........] duration: 3.510155083s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:19:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.51017484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................X............X.....X.X............X.......] duration: 3.46015779s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:19:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460192275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X...................X.....X......................] duration: 5.727714676s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:18:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.727744682s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................X............X.....X.X............X......] duration: 3.514630495s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:18:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514653379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X...................X.....X.....................] duration: 5.779175576s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:18:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.779198428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....................X............X.....X.X............X.....] duration: 3.472215357s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:17:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47223216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X...................X.....X....................] duration: 5.774115612s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:17:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.774142702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...................X.....X...................] duration: 5.738677461s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:16:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.738699133s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....................X............X.....X.X............X....] duration: 3.429150898s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:16:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429181435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...................X.....X..................] duration: 5.691565844s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:16:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.691589578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......................X............X.....X.X............X...] duration: 3.518412495s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:16:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518441241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...................X.....X.................] duration: 7.083863971s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:15:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 7.083891384s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X......................X............X.....X.X............X..] duration: 3.429060476s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:15:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429086955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X...................X.....X................] duration: 5.72212434s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:15:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.722151612s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X......................X............X.....X.X............X.] duration: 3.493722634s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:14:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493750137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X...................X.....X...............] duration: 5.749100097s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:14:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.749124834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X......................X............X.....X.X............X] duration: 5.001416293s trace: https://jaeger.dev.storj.io/trace/4ad7c653adedb94f FAIL (2025-12-25 16:14:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001441571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X...................X.....X..............] duration: 5.905724527s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:13:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.905756037s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X......................X............X.....X.X............] duration: 3.414012701s trace: https://jaeger.dev.storj.io/trace/1579d2e0c05cd02c FAIL (2025-12-25 16:13:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.41403344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X...................X.....X.............] duration: 5.730931181s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:13:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.73097338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...................X.....X............] duration: 5.914216198s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:12:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.914249449s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)