Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 336.635411ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:44:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 336.663124ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX....X.XX...X..X..XXXX....X....XXX....XXX.X......XXXX..X...] duration: 6.126723658s trace: https://jaeger.dev.storj.io/trace/f5918a38d524d3d FAIL (2026-01-02 01:44:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:40:56.822331072 (658h11m56.693414137s) [upload=576599]: Starting upload Jan 2 01:40:56.822473503 (658h11m56.693556558s) [upload=576599]: Done starting upload Jan 2 01:40:56.822696609 (658h11m56.693779674s) [upload=576599]: Uploading segments... Jan 2 01:40:56.823952750 (658h11m56.695035785s) [upload=576599]: Got next segment. Inline: false Jan 2 01:40:56.824034997 (658h11m56.695118052s) [upload=576599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:40:56.824044274 (658h11m56.695127339s) [upload=576599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:40:57.068922214 (658h11m56.940005259s) [seg_pos={0 0}, upload=576599]: Begin upload segment... Jan 2 01:40:57.069015452 (658h11m56.940098497s) [seg_pos={0 0}, upload=576599]: Done begin upload segment. Jan 2 01:40:57.069115573 (658h11m56.940198608s) [noise=true , address=145.79.227.7:4294 , piece_id=FTZZOAQ4PLDIKB4WTF7ZWBKM5OFZ4YUHRKBMYZ2CZQHPKTHK4VQQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.069140911 (658h11m56.940223956s) [noise=true , address=145.79.227.41:4086 , piece_id=JAT2DBJOEX5M2OADKVJLJY6MZ3334LC6DFFBSDE4K5OC4ZN2SEXA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.069237705 (658h11m56.94032075s) [noise=true , address=45.91.116.12:4227 , piece_id=2VCLKOGJCAWBDYGB66MFJV5L6BPFXITSONGHBSNKJXJVVCD7UVUQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.070075621 (658h11m56.941158666s) [noise=true , address=145.79.227.11:4732 , piece_id=2KITEWEITPPWNBUPWMEC3KYZGKETICIMLSCS4TS2Q6CTW5VS5LXQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.070706772 (658h11m56.941789817s) [noise=true , address=77.90.31.21:4880 , piece_id=VIFKEUMNWTRMNGZMUHTJJJJH7X2SO55TGVOPEU3V6HHW6DPFAGRA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.071174243 (658h11m56.942257288s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.072340333 (658h11m56.943423378s) [noise=true , address=77.90.31.11:4556 , piece_id=RNAI2BXPCESDT25EFGPVAJEJIPY7ZECB2YPMUZSMSJJRPWJJAPKQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.073260284 (658h11m56.944343319s) [noise=true , address=145.79.227.2:4115 , piece_id=YCYULB6RM6IFGU2WVD4JDBWZZWBBI5DELLR6PR2AY64UJQWJ6KHQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.073968874 (658h11m56.945051909s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.075763611 (658h11m56.946846656s) [noise=true , address=145.79.227.36:4518 , piece_id=K45OY6LITBIOYQJ7FVOXO6ADTFZWWUCDNQ354DSWM7JM4TIKEOWA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.076288050 (658h11m56.947371095s) [noise=true , address=45.91.116.14:4189 , piece_id=WOLP3B2IDCIOKMQJKG6D3DQ7QSBIXMVNDEHPZMM5YP7RSLIHH4IQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.077102981 (658h11m56.948186036s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.077860594 (658h11m56.948943639s) [noise=true , address=145.79.227.14:4839 , piece_id=OSVH46VUMMSEUWAAUITS4YFK4ITDPJGGD7KEASC3ZQ2LZ5WV66JA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.078296824 (658h11m56.949379869s) [noise=true , address=45.91.116.8:4048 , piece_id=44YHW7P7QJ62ATQ6KPI22LQGZZH2WZOHWZ3ZTSUIAUHCE2HCROUQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.078739728 (658h11m56.949822763s) [noise=true , address=145.79.227.3:4151 , piece_id=UWZ6E6STYZGZAXFHHVH3K5ZDEFZQG7FOIQPGFWJT2FSSZ4BVH5CQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.079175598 (658h11m56.950258643s) [noise=true , address=145.79.227.5:4221 , piece_id=IG3CLIFX4IS4OD6ZUNZ3J57PRFJ3N36RLSALOAJ427DGBIQEOPPA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:58.673183992 (658h11m58.544267027s) [upload=576599]: Next returned nil segment Jan 2 01:40:58.673225871 (658h11m58.544308917s) [upload=576599]: Waiting for error group managing segments... Jan 2 01:40:59.052157347 (658h11m58.923240382s) [noise=true , address=145.79.227.7:4294 , piece_id=FTZZOAQ4PLDIKB4WTF7ZWBKM5OFZ4YUHRKBMYZ2CZQHPKTHK4VQQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.052939716 (658h11m58.924022751s) [noise=true , address=145.79.227.2:4115 , piece_id=YCYULB6RM6IFGU2WVD4JDBWZZWBBI5DELLR6PR2AY64UJQWJ6KHQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.055676166 (658h11m58.926759211s) [noise=true , address=145.79.227.5:4221 , piece_id=IG3CLIFX4IS4OD6ZUNZ3J57PRFJ3N36RLSALOAJ427DGBIQEOPPA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.061877114 (658h11m58.932960149s) [noise=true , address=145.79.227.36:4518 , piece_id=K45OY6LITBIOYQJ7FVOXO6ADTFZWWUCDNQ354DSWM7JM4TIKEOWA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.069574159 (658h11m58.940657194s) [noise=true , address=145.79.227.3:4151 , piece_id=UWZ6E6STYZGZAXFHHVH3K5ZDEFZQG7FOIQPGFWJT2FSSZ4BVH5CQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.079231317 (658h11m58.950314352s) [noise=true , address=145.79.227.14:4839 , piece_id=OSVH46VUMMSEUWAAUITS4YFK4ITDPJGGD7KEASC3ZQ2LZ5WV66JA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.086823924 (658h11m58.957906969s) [noise=true , address=145.79.227.41:4086 , piece_id=JAT2DBJOEX5M2OADKVJLJY6MZ3334LC6DFFBSDE4K5OC4ZN2SEXA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.128987795 (658h11m59.00007085s) [noise=true , address=145.79.227.11:4732 , piece_id=2KITEWEITPPWNBUPWMEC3KYZGKETICIMLSCS4TS2Q6CTW5VS5LXQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.306482014 (658h11m59.177565059s) [noise=true , address=77.90.31.11:4556 , piece_id=RNAI2BXPCESDT25EFGPVAJEJIPY7ZECB2YPMUZSMSJJRPWJJAPKQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.367366371 (658h11m59.238449406s) [noise=true , address=77.90.31.21:4880 , piece_id=VIFKEUMNWTRMNGZMUHTJJJJH7X2SO55TGVOPEU3V6HHW6DPFAGRA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.368467327 (658h11m59.239550362s) [noise=true , address=45.91.116.8:4048 , piece_id=44YHW7P7QJ62ATQ6KPI22LQGZZH2WZOHWZ3ZTSUIAUHCE2HCROUQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.381075784 (658h11m59.252158829s) [noise=true , address=45.91.116.14:4189 , piece_id=WOLP3B2IDCIOKMQJKG6D3DQ7QSBIXMVNDEHPZMM5YP7RSLIHH4IQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.476010855 (658h11m59.34709389s) [noise=true , address=45.91.116.12:4227 , piece_id=2VCLKOGJCAWBDYGB66MFJV5L6BPFXITSONGHBSNKJXJVVCD7UVUQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.476029612 (658h11m59.347112647s) [seg_pos={0 0}, upload=576599]: Segment reached optimal threshold of 13 pieces. Jan 2 01:40:59.476243849 (658h11m59.347326884s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476251143 (658h11m59.347334188s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476504185 (658h11m59.34758722s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476511208 (658h11m59.347594243s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476630596 (658h11m59.347713631s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476637288 (658h11m59.347720323s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476667406 (658h11m59.347750441s) [seg_pos={0 0}, upload=576599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:40:59.476730536 (658h11m59.347813571s) [upload=576599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:40:59.476734784 (658h11m59.347817829s) [upload=576599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:40:59.764113388 (658h11m59.635196423s) [upload=576599]: Done uploading segments... Jan 2 01:40:59.764131031 (658h11m59.635214076s) [upload=576599]: Upload finished. err: <nil> Duration: 6.126751621s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.025423565s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:43:53 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 4.025496223s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 400.379505ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:43:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 400.404612ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.954092996s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:43:13 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.95413176s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......XXXX..X..] duration: 6.048082138s trace: https://jaeger.dev.storj.io/trace/f5918a38d524d3d FAIL (2026-01-02 01:43:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:40:56.822331072 (658h11m56.693414137s) [upload=576599]: Starting upload Jan 2 01:40:56.822473503 (658h11m56.693556558s) [upload=576599]: Done starting upload Jan 2 01:40:56.822696609 (658h11m56.693779674s) [upload=576599]: Uploading segments... Jan 2 01:40:56.823952750 (658h11m56.695035785s) [upload=576599]: Got next segment. Inline: false Jan 2 01:40:56.824034997 (658h11m56.695118052s) [upload=576599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:40:56.824044274 (658h11m56.695127339s) [upload=576599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:40:57.068922214 (658h11m56.940005259s) [seg_pos={0 0}, upload=576599]: Begin upload segment... Jan 2 01:40:57.069015452 (658h11m56.940098497s) [seg_pos={0 0}, upload=576599]: Done begin upload segment. Jan 2 01:40:57.069115573 (658h11m56.940198608s) [noise=true , address=145.79.227.7:4294 , piece_id=FTZZOAQ4PLDIKB4WTF7ZWBKM5OFZ4YUHRKBMYZ2CZQHPKTHK4VQQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.069140911 (658h11m56.940223956s) [noise=true , address=145.79.227.41:4086 , piece_id=JAT2DBJOEX5M2OADKVJLJY6MZ3334LC6DFFBSDE4K5OC4ZN2SEXA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.069237705 (658h11m56.94032075s) [noise=true , address=45.91.116.12:4227 , piece_id=2VCLKOGJCAWBDYGB66MFJV5L6BPFXITSONGHBSNKJXJVVCD7UVUQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.070075621 (658h11m56.941158666s) [noise=true , address=145.79.227.11:4732 , piece_id=2KITEWEITPPWNBUPWMEC3KYZGKETICIMLSCS4TS2Q6CTW5VS5LXQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.070706772 (658h11m56.941789817s) [noise=true , address=77.90.31.21:4880 , piece_id=VIFKEUMNWTRMNGZMUHTJJJJH7X2SO55TGVOPEU3V6HHW6DPFAGRA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.071174243 (658h11m56.942257288s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.072340333 (658h11m56.943423378s) [noise=true , address=77.90.31.11:4556 , piece_id=RNAI2BXPCESDT25EFGPVAJEJIPY7ZECB2YPMUZSMSJJRPWJJAPKQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.073260284 (658h11m56.944343319s) [noise=true , address=145.79.227.2:4115 , piece_id=YCYULB6RM6IFGU2WVD4JDBWZZWBBI5DELLR6PR2AY64UJQWJ6KHQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.073968874 (658h11m56.945051909s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.075763611 (658h11m56.946846656s) [noise=true , address=145.79.227.36:4518 , piece_id=K45OY6LITBIOYQJ7FVOXO6ADTFZWWUCDNQ354DSWM7JM4TIKEOWA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.076288050 (658h11m56.947371095s) [noise=true , address=45.91.116.14:4189 , piece_id=WOLP3B2IDCIOKMQJKG6D3DQ7QSBIXMVNDEHPZMM5YP7RSLIHH4IQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.077102981 (658h11m56.948186036s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.077860594 (658h11m56.948943639s) [noise=true , address=145.79.227.14:4839 , piece_id=OSVH46VUMMSEUWAAUITS4YFK4ITDPJGGD7KEASC3ZQ2LZ5WV66JA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.078296824 (658h11m56.949379869s) [noise=true , address=45.91.116.8:4048 , piece_id=44YHW7P7QJ62ATQ6KPI22LQGZZH2WZOHWZ3ZTSUIAUHCE2HCROUQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.078739728 (658h11m56.949822763s) [noise=true , address=145.79.227.3:4151 , piece_id=UWZ6E6STYZGZAXFHHVH3K5ZDEFZQG7FOIQPGFWJT2FSSZ4BVH5CQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.079175598 (658h11m56.950258643s) [noise=true , address=145.79.227.5:4221 , piece_id=IG3CLIFX4IS4OD6ZUNZ3J57PRFJ3N36RLSALOAJ427DGBIQEOPPA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:58.673183992 (658h11m58.544267027s) [upload=576599]: Next returned nil segment Jan 2 01:40:58.673225871 (658h11m58.544308917s) [upload=576599]: Waiting for error group managing segments... Jan 2 01:40:59.052157347 (658h11m58.923240382s) [noise=true , address=145.79.227.7:4294 , piece_id=FTZZOAQ4PLDIKB4WTF7ZWBKM5OFZ4YUHRKBMYZ2CZQHPKTHK4VQQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.052939716 (658h11m58.924022751s) [noise=true , address=145.79.227.2:4115 , piece_id=YCYULB6RM6IFGU2WVD4JDBWZZWBBI5DELLR6PR2AY64UJQWJ6KHQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.055676166 (658h11m58.926759211s) [noise=true , address=145.79.227.5:4221 , piece_id=IG3CLIFX4IS4OD6ZUNZ3J57PRFJ3N36RLSALOAJ427DGBIQEOPPA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.061877114 (658h11m58.932960149s) [noise=true , address=145.79.227.36:4518 , piece_id=K45OY6LITBIOYQJ7FVOXO6ADTFZWWUCDNQ354DSWM7JM4TIKEOWA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.069574159 (658h11m58.940657194s) [noise=true , address=145.79.227.3:4151 , piece_id=UWZ6E6STYZGZAXFHHVH3K5ZDEFZQG7FOIQPGFWJT2FSSZ4BVH5CQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.079231317 (658h11m58.950314352s) [noise=true , address=145.79.227.14:4839 , piece_id=OSVH46VUMMSEUWAAUITS4YFK4ITDPJGGD7KEASC3ZQ2LZ5WV66JA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.086823924 (658h11m58.957906969s) [noise=true , address=145.79.227.41:4086 , piece_id=JAT2DBJOEX5M2OADKVJLJY6MZ3334LC6DFFBSDE4K5OC4ZN2SEXA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.128987795 (658h11m59.00007085s) [noise=true , address=145.79.227.11:4732 , piece_id=2KITEWEITPPWNBUPWMEC3KYZGKETICIMLSCS4TS2Q6CTW5VS5LXQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.306482014 (658h11m59.177565059s) [noise=true , address=77.90.31.11:4556 , piece_id=RNAI2BXPCESDT25EFGPVAJEJIPY7ZECB2YPMUZSMSJJRPWJJAPKQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.367366371 (658h11m59.238449406s) [noise=true , address=77.90.31.21:4880 , piece_id=VIFKEUMNWTRMNGZMUHTJJJJH7X2SO55TGVOPEU3V6HHW6DPFAGRA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.368467327 (658h11m59.239550362s) [noise=true , address=45.91.116.8:4048 , piece_id=44YHW7P7QJ62ATQ6KPI22LQGZZH2WZOHWZ3ZTSUIAUHCE2HCROUQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.381075784 (658h11m59.252158829s) [noise=true , address=45.91.116.14:4189 , piece_id=WOLP3B2IDCIOKMQJKG6D3DQ7QSBIXMVNDEHPZMM5YP7RSLIHH4IQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.476010855 (658h11m59.34709389s) [noise=true , address=45.91.116.12:4227 , piece_id=2VCLKOGJCAWBDYGB66MFJV5L6BPFXITSONGHBSNKJXJVVCD7UVUQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.476029612 (658h11m59.347112647s) [seg_pos={0 0}, upload=576599]: Segment reached optimal threshold of 13 pieces. Jan 2 01:40:59.476243849 (658h11m59.347326884s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476251143 (658h11m59.347334188s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476504185 (658h11m59.34758722s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476511208 (658h11m59.347594243s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476630596 (658h11m59.347713631s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476637288 (658h11m59.347720323s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476667406 (658h11m59.347750441s) [seg_pos={0 0}, upload=576599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:40:59.476730536 (658h11m59.347813571s) [upload=576599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:40:59.476734784 (658h11m59.347817829s) [upload=576599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:40:59.764113388 (658h11m59.635196423s) [upload=576599]: Done uploading segments... Jan 2 01:40:59.764131031 (658h11m59.635214076s) [upload=576599]: Upload finished. err: <nil> Duration: 6.048112937s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 211.492221ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:43:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 211.510566ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 149.854881ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:42:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 149.883405ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.230830538s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:42:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.230855906s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......XXXX..X.] duration: 6.057619811s trace: https://jaeger.dev.storj.io/trace/f5918a38d524d3d FAIL (2026-01-02 01:42:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:40:56.822331072 (658h11m56.693414137s) [upload=576599]: Starting upload Jan 2 01:40:56.822473503 (658h11m56.693556558s) [upload=576599]: Done starting upload Jan 2 01:40:56.822696609 (658h11m56.693779674s) [upload=576599]: Uploading segments... Jan 2 01:40:56.823952750 (658h11m56.695035785s) [upload=576599]: Got next segment. Inline: false Jan 2 01:40:56.824034997 (658h11m56.695118052s) [upload=576599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:40:56.824044274 (658h11m56.695127339s) [upload=576599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:40:57.068922214 (658h11m56.940005259s) [seg_pos={0 0}, upload=576599]: Begin upload segment... Jan 2 01:40:57.069015452 (658h11m56.940098497s) [seg_pos={0 0}, upload=576599]: Done begin upload segment. Jan 2 01:40:57.069115573 (658h11m56.940198608s) [noise=true , address=145.79.227.7:4294 , piece_id=FTZZOAQ4PLDIKB4WTF7ZWBKM5OFZ4YUHRKBMYZ2CZQHPKTHK4VQQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.069140911 (658h11m56.940223956s) [noise=true , address=145.79.227.41:4086 , piece_id=JAT2DBJOEX5M2OADKVJLJY6MZ3334LC6DFFBSDE4K5OC4ZN2SEXA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.069237705 (658h11m56.94032075s) [noise=true , address=45.91.116.12:4227 , piece_id=2VCLKOGJCAWBDYGB66MFJV5L6BPFXITSONGHBSNKJXJVVCD7UVUQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.070075621 (658h11m56.941158666s) [noise=true , address=145.79.227.11:4732 , piece_id=2KITEWEITPPWNBUPWMEC3KYZGKETICIMLSCS4TS2Q6CTW5VS5LXQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.070706772 (658h11m56.941789817s) [noise=true , address=77.90.31.21:4880 , piece_id=VIFKEUMNWTRMNGZMUHTJJJJH7X2SO55TGVOPEU3V6HHW6DPFAGRA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.071174243 (658h11m56.942257288s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.072340333 (658h11m56.943423378s) [noise=true , address=77.90.31.11:4556 , piece_id=RNAI2BXPCESDT25EFGPVAJEJIPY7ZECB2YPMUZSMSJJRPWJJAPKQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.073260284 (658h11m56.944343319s) [noise=true , address=145.79.227.2:4115 , piece_id=YCYULB6RM6IFGU2WVD4JDBWZZWBBI5DELLR6PR2AY64UJQWJ6KHQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.073968874 (658h11m56.945051909s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.075763611 (658h11m56.946846656s) [noise=true , address=145.79.227.36:4518 , piece_id=K45OY6LITBIOYQJ7FVOXO6ADTFZWWUCDNQ354DSWM7JM4TIKEOWA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.076288050 (658h11m56.947371095s) [noise=true , address=45.91.116.14:4189 , piece_id=WOLP3B2IDCIOKMQJKG6D3DQ7QSBIXMVNDEHPZMM5YP7RSLIHH4IQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.077102981 (658h11m56.948186036s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.077860594 (658h11m56.948943639s) [noise=true , address=145.79.227.14:4839 , piece_id=OSVH46VUMMSEUWAAUITS4YFK4ITDPJGGD7KEASC3ZQ2LZ5WV66JA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.078296824 (658h11m56.949379869s) [noise=true , address=45.91.116.8:4048 , piece_id=44YHW7P7QJ62ATQ6KPI22LQGZZH2WZOHWZ3ZTSUIAUHCE2HCROUQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.078739728 (658h11m56.949822763s) [noise=true , address=145.79.227.3:4151 , piece_id=UWZ6E6STYZGZAXFHHVH3K5ZDEFZQG7FOIQPGFWJT2FSSZ4BVH5CQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.079175598 (658h11m56.950258643s) [noise=true , address=145.79.227.5:4221 , piece_id=IG3CLIFX4IS4OD6ZUNZ3J57PRFJ3N36RLSALOAJ427DGBIQEOPPA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:58.673183992 (658h11m58.544267027s) [upload=576599]: Next returned nil segment Jan 2 01:40:58.673225871 (658h11m58.544308917s) [upload=576599]: Waiting for error group managing segments... Jan 2 01:40:59.052157347 (658h11m58.923240382s) [noise=true , address=145.79.227.7:4294 , piece_id=FTZZOAQ4PLDIKB4WTF7ZWBKM5OFZ4YUHRKBMYZ2CZQHPKTHK4VQQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.052939716 (658h11m58.924022751s) [noise=true , address=145.79.227.2:4115 , piece_id=YCYULB6RM6IFGU2WVD4JDBWZZWBBI5DELLR6PR2AY64UJQWJ6KHQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.055676166 (658h11m58.926759211s) [noise=true , address=145.79.227.5:4221 , piece_id=IG3CLIFX4IS4OD6ZUNZ3J57PRFJ3N36RLSALOAJ427DGBIQEOPPA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.061877114 (658h11m58.932960149s) [noise=true , address=145.79.227.36:4518 , piece_id=K45OY6LITBIOYQJ7FVOXO6ADTFZWWUCDNQ354DSWM7JM4TIKEOWA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.069574159 (658h11m58.940657194s) [noise=true , address=145.79.227.3:4151 , piece_id=UWZ6E6STYZGZAXFHHVH3K5ZDEFZQG7FOIQPGFWJT2FSSZ4BVH5CQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.079231317 (658h11m58.950314352s) [noise=true , address=145.79.227.14:4839 , piece_id=OSVH46VUMMSEUWAAUITS4YFK4ITDPJGGD7KEASC3ZQ2LZ5WV66JA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.086823924 (658h11m58.957906969s) [noise=true , address=145.79.227.41:4086 , piece_id=JAT2DBJOEX5M2OADKVJLJY6MZ3334LC6DFFBSDE4K5OC4ZN2SEXA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.128987795 (658h11m59.00007085s) [noise=true , address=145.79.227.11:4732 , piece_id=2KITEWEITPPWNBUPWMEC3KYZGKETICIMLSCS4TS2Q6CTW5VS5LXQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.306482014 (658h11m59.177565059s) [noise=true , address=77.90.31.11:4556 , piece_id=RNAI2BXPCESDT25EFGPVAJEJIPY7ZECB2YPMUZSMSJJRPWJJAPKQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.367366371 (658h11m59.238449406s) [noise=true , address=77.90.31.21:4880 , piece_id=VIFKEUMNWTRMNGZMUHTJJJJH7X2SO55TGVOPEU3V6HHW6DPFAGRA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.368467327 (658h11m59.239550362s) [noise=true , address=45.91.116.8:4048 , piece_id=44YHW7P7QJ62ATQ6KPI22LQGZZH2WZOHWZ3ZTSUIAUHCE2HCROUQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.381075784 (658h11m59.252158829s) [noise=true , address=45.91.116.14:4189 , piece_id=WOLP3B2IDCIOKMQJKG6D3DQ7QSBIXMVNDEHPZMM5YP7RSLIHH4IQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.476010855 (658h11m59.34709389s) [noise=true , address=45.91.116.12:4227 , piece_id=2VCLKOGJCAWBDYGB66MFJV5L6BPFXITSONGHBSNKJXJVVCD7UVUQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.476029612 (658h11m59.347112647s) [seg_pos={0 0}, upload=576599]: Segment reached optimal threshold of 13 pieces. Jan 2 01:40:59.476243849 (658h11m59.347326884s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476251143 (658h11m59.347334188s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476504185 (658h11m59.34758722s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476511208 (658h11m59.347594243s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476630596 (658h11m59.347713631s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476637288 (658h11m59.347720323s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476667406 (658h11m59.347750441s) [seg_pos={0 0}, upload=576599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:40:59.476730536 (658h11m59.347813571s) [upload=576599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:40:59.476734784 (658h11m59.347817829s) [upload=576599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:40:59.764113388 (658h11m59.635196423s) [upload=576599]: Done uploading segments... Jan 2 01:40:59.764131031 (658h11m59.635214076s) [upload=576599]: Upload finished. err: <nil> Duration: 6.05765158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......XXXX..X] duration: 30.002493124s trace: https://jaeger.dev.storj.io/trace/f5918a38d524d3d FAIL (2026-01-02 01:41:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:40:56.822331072 (658h11m56.693414137s) [upload=576599]: Starting upload Jan 2 01:40:56.822473503 (658h11m56.693556558s) [upload=576599]: Done starting upload Jan 2 01:40:56.822696609 (658h11m56.693779674s) [upload=576599]: Uploading segments... Jan 2 01:40:56.823952750 (658h11m56.695035785s) [upload=576599]: Got next segment. Inline: false Jan 2 01:40:56.824034997 (658h11m56.695118052s) [upload=576599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:40:56.824044274 (658h11m56.695127339s) [upload=576599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:40:57.068922214 (658h11m56.940005259s) [seg_pos={0 0}, upload=576599]: Begin upload segment... Jan 2 01:40:57.069015452 (658h11m56.940098497s) [seg_pos={0 0}, upload=576599]: Done begin upload segment. Jan 2 01:40:57.069115573 (658h11m56.940198608s) [noise=true , address=145.79.227.7:4294 , piece_id=FTZZOAQ4PLDIKB4WTF7ZWBKM5OFZ4YUHRKBMYZ2CZQHPKTHK4VQQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.069140911 (658h11m56.940223956s) [noise=true , address=145.79.227.41:4086 , piece_id=JAT2DBJOEX5M2OADKVJLJY6MZ3334LC6DFFBSDE4K5OC4ZN2SEXA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.069237705 (658h11m56.94032075s) [noise=true , address=45.91.116.12:4227 , piece_id=2VCLKOGJCAWBDYGB66MFJV5L6BPFXITSONGHBSNKJXJVVCD7UVUQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.070075621 (658h11m56.941158666s) [noise=true , address=145.79.227.11:4732 , piece_id=2KITEWEITPPWNBUPWMEC3KYZGKETICIMLSCS4TS2Q6CTW5VS5LXQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.070706772 (658h11m56.941789817s) [noise=true , address=77.90.31.21:4880 , piece_id=VIFKEUMNWTRMNGZMUHTJJJJH7X2SO55TGVOPEU3V6HHW6DPFAGRA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.071174243 (658h11m56.942257288s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.072340333 (658h11m56.943423378s) [noise=true , address=77.90.31.11:4556 , piece_id=RNAI2BXPCESDT25EFGPVAJEJIPY7ZECB2YPMUZSMSJJRPWJJAPKQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.073260284 (658h11m56.944343319s) [noise=true , address=145.79.227.2:4115 , piece_id=YCYULB6RM6IFGU2WVD4JDBWZZWBBI5DELLR6PR2AY64UJQWJ6KHQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.073968874 (658h11m56.945051909s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.075763611 (658h11m56.946846656s) [noise=true , address=145.79.227.36:4518 , piece_id=K45OY6LITBIOYQJ7FVOXO6ADTFZWWUCDNQ354DSWM7JM4TIKEOWA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.076288050 (658h11m56.947371095s) [noise=true , address=45.91.116.14:4189 , piece_id=WOLP3B2IDCIOKMQJKG6D3DQ7QSBIXMVNDEHPZMM5YP7RSLIHH4IQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.077102981 (658h11m56.948186036s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.077860594 (658h11m56.948943639s) [noise=true , address=145.79.227.14:4839 , piece_id=OSVH46VUMMSEUWAAUITS4YFK4ITDPJGGD7KEASC3ZQ2LZ5WV66JA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.078296824 (658h11m56.949379869s) [noise=true , address=45.91.116.8:4048 , piece_id=44YHW7P7QJ62ATQ6KPI22LQGZZH2WZOHWZ3ZTSUIAUHCE2HCROUQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.078739728 (658h11m56.949822763s) [noise=true , address=145.79.227.3:4151 , piece_id=UWZ6E6STYZGZAXFHHVH3K5ZDEFZQG7FOIQPGFWJT2FSSZ4BVH5CQ, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:57.079175598 (658h11m56.950258643s) [noise=true , address=145.79.227.5:4221 , piece_id=IG3CLIFX4IS4OD6ZUNZ3J57PRFJ3N36RLSALOAJ427DGBIQEOPPA, seg_pos={0 0}, upload=576599]: Uploading piece... Jan 2 01:40:58.673183992 (658h11m58.544267027s) [upload=576599]: Next returned nil segment Jan 2 01:40:58.673225871 (658h11m58.544308917s) [upload=576599]: Waiting for error group managing segments... Jan 2 01:40:59.052157347 (658h11m58.923240382s) [noise=true , address=145.79.227.7:4294 , piece_id=FTZZOAQ4PLDIKB4WTF7ZWBKM5OFZ4YUHRKBMYZ2CZQHPKTHK4VQQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.052939716 (658h11m58.924022751s) [noise=true , address=145.79.227.2:4115 , piece_id=YCYULB6RM6IFGU2WVD4JDBWZZWBBI5DELLR6PR2AY64UJQWJ6KHQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.055676166 (658h11m58.926759211s) [noise=true , address=145.79.227.5:4221 , piece_id=IG3CLIFX4IS4OD6ZUNZ3J57PRFJ3N36RLSALOAJ427DGBIQEOPPA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.061877114 (658h11m58.932960149s) [noise=true , address=145.79.227.36:4518 , piece_id=K45OY6LITBIOYQJ7FVOXO6ADTFZWWUCDNQ354DSWM7JM4TIKEOWA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.069574159 (658h11m58.940657194s) [noise=true , address=145.79.227.3:4151 , piece_id=UWZ6E6STYZGZAXFHHVH3K5ZDEFZQG7FOIQPGFWJT2FSSZ4BVH5CQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.079231317 (658h11m58.950314352s) [noise=true , address=145.79.227.14:4839 , piece_id=OSVH46VUMMSEUWAAUITS4YFK4ITDPJGGD7KEASC3ZQ2LZ5WV66JA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.086823924 (658h11m58.957906969s) [noise=true , address=145.79.227.41:4086 , piece_id=JAT2DBJOEX5M2OADKVJLJY6MZ3334LC6DFFBSDE4K5OC4ZN2SEXA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.128987795 (658h11m59.00007085s) [noise=true , address=145.79.227.11:4732 , piece_id=2KITEWEITPPWNBUPWMEC3KYZGKETICIMLSCS4TS2Q6CTW5VS5LXQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.306482014 (658h11m59.177565059s) [noise=true , address=77.90.31.11:4556 , piece_id=RNAI2BXPCESDT25EFGPVAJEJIPY7ZECB2YPMUZSMSJJRPWJJAPKQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.367366371 (658h11m59.238449406s) [noise=true , address=77.90.31.21:4880 , piece_id=VIFKEUMNWTRMNGZMUHTJJJJH7X2SO55TGVOPEU3V6HHW6DPFAGRA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.368467327 (658h11m59.239550362s) [noise=true , address=45.91.116.8:4048 , piece_id=44YHW7P7QJ62ATQ6KPI22LQGZZH2WZOHWZ3ZTSUIAUHCE2HCROUQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.381075784 (658h11m59.252158829s) [noise=true , address=45.91.116.14:4189 , piece_id=WOLP3B2IDCIOKMQJKG6D3DQ7QSBIXMVNDEHPZMM5YP7RSLIHH4IQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.476010855 (658h11m59.34709389s) [noise=true , address=45.91.116.12:4227 , piece_id=2VCLKOGJCAWBDYGB66MFJV5L6BPFXITSONGHBSNKJXJVVCD7UVUQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: <nil> Jan 2 01:40:59.476029612 (658h11m59.347112647s) [seg_pos={0 0}, upload=576599]: Segment reached optimal threshold of 13 pieces. Jan 2 01:40:59.476243849 (658h11m59.347326884s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476251143 (658h11m59.347334188s) [noise=true , address=45.91.116.9:4263 , piece_id=XNH6DFWDU2LWLODCJNCDBI3CVADEQCONFSITYVOZEFDCS4BTCFMQ, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476504185 (658h11m59.34758722s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476511208 (658h11m59.347594243s) [noise=true , address=45.91.116.6:4976 , piece_id=PT2VLJ4G4J2QS327TODOBPUJBE4TS46ETWEX4ARPTXIBPAX2G3UA, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476630596 (658h11m59.347713631s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Jan 2 01:40:59.476637288 (658h11m59.347720323s) [noise=true , address=77.90.31.12:4590 , piece_id=LHTC4J6ZPINJIXZ7IXJE5IHZZUCP7DXNI36CDAH6Y5OTKBILQUEA, seg_pos={0 0}, upload=576599]: Long tail cancellation due to optimal threshold Jan 2 01:40:59.476667406 (658h11m59.347750441s) [seg_pos={0 0}, upload=576599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:40:59.476730536 (658h11m59.347813571s) [upload=576599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:40:59.476734784 (658h11m59.347817829s) [upload=576599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:40:59.764113388 (658h11m59.635196423s) [upload=576599]: Done uploading segments... Jan 2 01:40:59.764131031 (658h11m59.635214076s) [upload=576599]: Upload finished. err: <nil> Duration: 30.00251774s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 193.010172ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:41:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 193.04095ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 3.344731351s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:40:59 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.344763031s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 148.89759ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:40:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 148.930932ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......XXXX..] duration: 29.966959625s trace: https://jaeger.dev.storj.io/trace/24866768d6f6be61 FAIL (2026-01-02 01:40:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:38:12.390378712 (658h9m12.261461757s) [upload=576561]: Starting upload Jan 2 01:38:12.390483982 (658h9m12.261567027s) [upload=576561]: Done starting upload Jan 2 01:38:12.390523227 (658h9m12.261606292s) [upload=576561]: Uploading segments... Jan 2 01:38:12.390793081 (658h9m12.261876136s) [upload=576561]: Got next segment. Inline: false Jan 2 01:38:12.390832666 (658h9m12.261915711s) [upload=576561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:38:12.390838527 (658h9m12.261921582s) [upload=576561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:38:12.642804033 (658h9m12.513887068s) [seg_pos={0 0}, upload=576561]: Begin upload segment... Jan 2 01:38:12.642863557 (658h9m12.513946602s) [seg_pos={0 0}, upload=576561]: Done begin upload segment. Jan 2 01:38:12.642892541 (658h9m12.513975576s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.642902250 (658h9m12.513985295s) [noise=true , address=145.79.227.46:4602 , piece_id=4M4GWXMLUKKH7NV6Q23KZGKLZAUPOJYMFGJL2IDWFQWDBJ2GG45Q, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.643445063 (658h9m12.514528098s) [noise=true , address=145.79.227.50:4375 , piece_id=SDTNWJQ3PWKHFJYQZ5TQPR3U4L2KXLK5RLMGNAJMACJCI2ZA2XEQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.643876686 (658h9m12.514959721s) [noise=true , address=145.79.227.14:4838 , piece_id=PJV3IGOVX4BNBQQ7FXZWU7AHA4TBSDQOW5EYNAIER45ONTG3GI4Q, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.644285343 (658h9m12.515368388s) [noise=true , address=45.91.116.12:4227 , piece_id=4G7QXQDWOJJQOK22PNEWAP4HHVOV7T5FVHHXGUC3LUVQOFP6PLRA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.644699863 (658h9m12.515782898s) [noise=true , address=77.90.31.15:4700 , piece_id=WQX2OA6LHZREKK7N2UR3JXPUPVLR3OUAPMQLLV5BAO3EALFTZFHQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645008901 (658h9m12.516091946s) [noise=true , address=145.79.227.38:4978 , piece_id=O7LPFKZSOEZSQAJGD2GISKXW6DCQTVRHH4NL2PZJE2IKP7ZP5WQQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645338007 (658h9m12.516421042s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645732098 (658h9m12.516815144s) [noise=true , address=145.79.227.18:4006 , piece_id=7OULP7JYSHUH7VMKBSLXSHL67Y4SFLXTCG44MVIAD3OSP5D6SDBA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.647470698 (658h9m12.518553743s) [noise=true , address=145.79.227.10:4401 , piece_id=CWO7NJO5KRU5AA3BLVGHLKV5CFNLQLZOJBQV3GYZDLKDSJ77Y7HA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.647639169 (658h9m12.518722214s) [noise=true , address=45.91.116.13:4153 , piece_id=BRAGSIR4JZYNLV3VVH2GS6IQQHTYJWNOVOLP5FX3GUXAU3SQGOZA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.649109919 (658h9m12.520192964s) [noise=true , address=77.90.31.10:4517 , piece_id=QK6A5K4H3ALLRWYJ42YPQSYDYTGNM6QLI4DBZGVTCKMBR6BSELZA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.649461709 (658h9m12.520544754s) [noise=true , address=145.79.227.42:4122 , piece_id=FAZUQRDBMJ4GMW5VFXR7UF37UCSYFOVFGAGYB76FYU46EJE7A7YQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.651511643 (658h9m12.522594688s) [noise=true , address=145.79.227.4:4187 , piece_id=R76LWRLP3ANDMWLOADX4IUBVWA4JVXX7N2XLYBHQDP6KD4VP2VIA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.652031503 (658h9m12.523114548s) [noise=true , address=145.79.227.11:4732 , piece_id=SVVNDZB32RH7I43E3DSL2IQQ3QTW5DVINOHC6VDP64MSHEOG2QTQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.653941179 (658h9m12.525024234s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:14.347536041 (658h9m14.218619076s) [upload=576561]: Next returned nil segment Jan 2 01:38:14.347553323 (658h9m14.218636369s) [upload=576561]: Waiting for error group managing segments... Jan 2 01:38:14.626656891 (658h9m14.497739926s) [noise=true , address=145.79.227.46:4602 , piece_id=4M4GWXMLUKKH7NV6Q23KZGKLZAUPOJYMFGJL2IDWFQWDBJ2GG45Q, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.639934913 (658h9m14.511017958s) [noise=true , address=145.79.227.18:4006 , piece_id=7OULP7JYSHUH7VMKBSLXSHL67Y4SFLXTCG44MVIAD3OSP5D6SDBA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.640164981 (658h9m14.511248016s) [noise=true , address=145.79.227.50:4375 , piece_id=SDTNWJQ3PWKHFJYQZ5TQPR3U4L2KXLK5RLMGNAJMACJCI2ZA2XEQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.640697506 (658h9m14.511780541s) [noise=true , address=145.79.227.10:4401 , piece_id=CWO7NJO5KRU5AA3BLVGHLKV5CFNLQLZOJBQV3GYZDLKDSJ77Y7HA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.641188240 (658h9m14.512271285s) [noise=true , address=145.79.227.14:4838 , piece_id=PJV3IGOVX4BNBQQ7FXZWU7AHA4TBSDQOW5EYNAIER45ONTG3GI4Q, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.649071289 (658h9m14.520154324s) [noise=true , address=145.79.227.4:4187 , piece_id=R76LWRLP3ANDMWLOADX4IUBVWA4JVXX7N2XLYBHQDP6KD4VP2VIA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.663599412 (658h9m14.534682457s) [noise=true , address=145.79.227.42:4122 , piece_id=FAZUQRDBMJ4GMW5VFXR7UF37UCSYFOVFGAGYB76FYU46EJE7A7YQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.697036497 (658h9m14.568119542s) [noise=true , address=145.79.227.38:4978 , piece_id=O7LPFKZSOEZSQAJGD2GISKXW6DCQTVRHH4NL2PZJE2IKP7ZP5WQQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.728208401 (658h9m14.599291446s) [noise=true , address=145.79.227.11:4732 , piece_id=SVVNDZB32RH7I43E3DSL2IQQ3QTW5DVINOHC6VDP64MSHEOG2QTQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.893700516 (658h9m14.764783561s) [noise=true , address=77.90.31.10:4517 , piece_id=QK6A5K4H3ALLRWYJ42YPQSYDYTGNM6QLI4DBZGVTCKMBR6BSELZA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.912641551 (658h9m14.783724596s) [noise=true , address=77.90.31.15:4700 , piece_id=WQX2OA6LHZREKK7N2UR3JXPUPVLR3OUAPMQLLV5BAO3EALFTZFHQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.942977492 (658h9m14.814060527s) [noise=true , address=45.91.116.12:4227 , piece_id=4G7QXQDWOJJQOK22PNEWAP4HHVOV7T5FVHHXGUC3LUVQOFP6PLRA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.966907175 (658h9m14.83799021s) [noise=true , address=45.91.116.13:4153 , piece_id=BRAGSIR4JZYNLV3VVH2GS6IQQHTYJWNOVOLP5FX3GUXAU3SQGOZA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.966926713 (658h9m14.838009758s) [seg_pos={0 0}, upload=576561]: Segment reached optimal threshold of 13 pieces. Jan 2 01:38:14.967106485 (658h9m14.83818953s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 2 01:38:14.967113628 (658h9m14.838196663s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967290826 (658h9m14.838373871s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:35442->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 2 01:38:14.967301647 (658h9m14.838384682s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967346632 (658h9m14.838429667s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 2 01:38:14.967356381 (658h9m14.838439436s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967402248 (658h9m14.838485293s) [seg_pos={0 0}, upload=576561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:38:14.967479115 (658h9m14.83856215s) [upload=576561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:38:14.967485477 (658h9m14.838568522s) [upload=576561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:38:15.263533106 (658h9m15.134616141s) [upload=576561]: Done uploading segments... Jan 2 01:38:15.263557072 (658h9m15.134640107s) [upload=576561]: Upload finished. err: <nil> Duration: 29.96698878s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 3.570532769s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:40:07 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.570555992s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 228.821069ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:39:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 228.845364ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......XXXX.] duration: 6.007443723s trace: https://jaeger.dev.storj.io/trace/24866768d6f6be61 FAIL (2026-01-02 01:38:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:38:12.390378712 (658h9m12.261461757s) [upload=576561]: Starting upload Jan 2 01:38:12.390483982 (658h9m12.261567027s) [upload=576561]: Done starting upload Jan 2 01:38:12.390523227 (658h9m12.261606292s) [upload=576561]: Uploading segments... Jan 2 01:38:12.390793081 (658h9m12.261876136s) [upload=576561]: Got next segment. Inline: false Jan 2 01:38:12.390832666 (658h9m12.261915711s) [upload=576561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:38:12.390838527 (658h9m12.261921582s) [upload=576561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:38:12.642804033 (658h9m12.513887068s) [seg_pos={0 0}, upload=576561]: Begin upload segment... Jan 2 01:38:12.642863557 (658h9m12.513946602s) [seg_pos={0 0}, upload=576561]: Done begin upload segment. Jan 2 01:38:12.642892541 (658h9m12.513975576s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.642902250 (658h9m12.513985295s) [noise=true , address=145.79.227.46:4602 , piece_id=4M4GWXMLUKKH7NV6Q23KZGKLZAUPOJYMFGJL2IDWFQWDBJ2GG45Q, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.643445063 (658h9m12.514528098s) [noise=true , address=145.79.227.50:4375 , piece_id=SDTNWJQ3PWKHFJYQZ5TQPR3U4L2KXLK5RLMGNAJMACJCI2ZA2XEQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.643876686 (658h9m12.514959721s) [noise=true , address=145.79.227.14:4838 , piece_id=PJV3IGOVX4BNBQQ7FXZWU7AHA4TBSDQOW5EYNAIER45ONTG3GI4Q, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.644285343 (658h9m12.515368388s) [noise=true , address=45.91.116.12:4227 , piece_id=4G7QXQDWOJJQOK22PNEWAP4HHVOV7T5FVHHXGUC3LUVQOFP6PLRA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.644699863 (658h9m12.515782898s) [noise=true , address=77.90.31.15:4700 , piece_id=WQX2OA6LHZREKK7N2UR3JXPUPVLR3OUAPMQLLV5BAO3EALFTZFHQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645008901 (658h9m12.516091946s) [noise=true , address=145.79.227.38:4978 , piece_id=O7LPFKZSOEZSQAJGD2GISKXW6DCQTVRHH4NL2PZJE2IKP7ZP5WQQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645338007 (658h9m12.516421042s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645732098 (658h9m12.516815144s) [noise=true , address=145.79.227.18:4006 , piece_id=7OULP7JYSHUH7VMKBSLXSHL67Y4SFLXTCG44MVIAD3OSP5D6SDBA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.647470698 (658h9m12.518553743s) [noise=true , address=145.79.227.10:4401 , piece_id=CWO7NJO5KRU5AA3BLVGHLKV5CFNLQLZOJBQV3GYZDLKDSJ77Y7HA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.647639169 (658h9m12.518722214s) [noise=true , address=45.91.116.13:4153 , piece_id=BRAGSIR4JZYNLV3VVH2GS6IQQHTYJWNOVOLP5FX3GUXAU3SQGOZA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.649109919 (658h9m12.520192964s) [noise=true , address=77.90.31.10:4517 , piece_id=QK6A5K4H3ALLRWYJ42YPQSYDYTGNM6QLI4DBZGVTCKMBR6BSELZA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.649461709 (658h9m12.520544754s) [noise=true , address=145.79.227.42:4122 , piece_id=FAZUQRDBMJ4GMW5VFXR7UF37UCSYFOVFGAGYB76FYU46EJE7A7YQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.651511643 (658h9m12.522594688s) [noise=true , address=145.79.227.4:4187 , piece_id=R76LWRLP3ANDMWLOADX4IUBVWA4JVXX7N2XLYBHQDP6KD4VP2VIA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.652031503 (658h9m12.523114548s) [noise=true , address=145.79.227.11:4732 , piece_id=SVVNDZB32RH7I43E3DSL2IQQ3QTW5DVINOHC6VDP64MSHEOG2QTQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.653941179 (658h9m12.525024234s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:14.347536041 (658h9m14.218619076s) [upload=576561]: Next returned nil segment Jan 2 01:38:14.347553323 (658h9m14.218636369s) [upload=576561]: Waiting for error group managing segments... Jan 2 01:38:14.626656891 (658h9m14.497739926s) [noise=true , address=145.79.227.46:4602 , piece_id=4M4GWXMLUKKH7NV6Q23KZGKLZAUPOJYMFGJL2IDWFQWDBJ2GG45Q, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.639934913 (658h9m14.511017958s) [noise=true , address=145.79.227.18:4006 , piece_id=7OULP7JYSHUH7VMKBSLXSHL67Y4SFLXTCG44MVIAD3OSP5D6SDBA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.640164981 (658h9m14.511248016s) [noise=true , address=145.79.227.50:4375 , piece_id=SDTNWJQ3PWKHFJYQZ5TQPR3U4L2KXLK5RLMGNAJMACJCI2ZA2XEQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.640697506 (658h9m14.511780541s) [noise=true , address=145.79.227.10:4401 , piece_id=CWO7NJO5KRU5AA3BLVGHLKV5CFNLQLZOJBQV3GYZDLKDSJ77Y7HA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.641188240 (658h9m14.512271285s) [noise=true , address=145.79.227.14:4838 , piece_id=PJV3IGOVX4BNBQQ7FXZWU7AHA4TBSDQOW5EYNAIER45ONTG3GI4Q, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.649071289 (658h9m14.520154324s) [noise=true , address=145.79.227.4:4187 , piece_id=R76LWRLP3ANDMWLOADX4IUBVWA4JVXX7N2XLYBHQDP6KD4VP2VIA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.663599412 (658h9m14.534682457s) [noise=true , address=145.79.227.42:4122 , piece_id=FAZUQRDBMJ4GMW5VFXR7UF37UCSYFOVFGAGYB76FYU46EJE7A7YQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.697036497 (658h9m14.568119542s) [noise=true , address=145.79.227.38:4978 , piece_id=O7LPFKZSOEZSQAJGD2GISKXW6DCQTVRHH4NL2PZJE2IKP7ZP5WQQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.728208401 (658h9m14.599291446s) [noise=true , address=145.79.227.11:4732 , piece_id=SVVNDZB32RH7I43E3DSL2IQQ3QTW5DVINOHC6VDP64MSHEOG2QTQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.893700516 (658h9m14.764783561s) [noise=true , address=77.90.31.10:4517 , piece_id=QK6A5K4H3ALLRWYJ42YPQSYDYTGNM6QLI4DBZGVTCKMBR6BSELZA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.912641551 (658h9m14.783724596s) [noise=true , address=77.90.31.15:4700 , piece_id=WQX2OA6LHZREKK7N2UR3JXPUPVLR3OUAPMQLLV5BAO3EALFTZFHQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.942977492 (658h9m14.814060527s) [noise=true , address=45.91.116.12:4227 , piece_id=4G7QXQDWOJJQOK22PNEWAP4HHVOV7T5FVHHXGUC3LUVQOFP6PLRA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.966907175 (658h9m14.83799021s) [noise=true , address=45.91.116.13:4153 , piece_id=BRAGSIR4JZYNLV3VVH2GS6IQQHTYJWNOVOLP5FX3GUXAU3SQGOZA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.966926713 (658h9m14.838009758s) [seg_pos={0 0}, upload=576561]: Segment reached optimal threshold of 13 pieces. Jan 2 01:38:14.967106485 (658h9m14.83818953s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 2 01:38:14.967113628 (658h9m14.838196663s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967290826 (658h9m14.838373871s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:35442->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 2 01:38:14.967301647 (658h9m14.838384682s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967346632 (658h9m14.838429667s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 2 01:38:14.967356381 (658h9m14.838439436s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967402248 (658h9m14.838485293s) [seg_pos={0 0}, upload=576561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:38:14.967479115 (658h9m14.83856215s) [upload=576561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:38:14.967485477 (658h9m14.838568522s) [upload=576561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:38:15.263533106 (658h9m15.134616141s) [upload=576561]: Done uploading segments... Jan 2 01:38:15.263557072 (658h9m15.134640107s) [upload=576561]: Upload finished. err: <nil> Duration: 6.007472317s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 3.408347489s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:38:59 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.408375503s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 193.895617ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:38:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 193.92828ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......XXXX] duration: 30.003394501s trace: https://jaeger.dev.storj.io/trace/24866768d6f6be61 FAIL (2026-01-02 01:38:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:38:12.390378712 (658h9m12.261461757s) [upload=576561]: Starting upload Jan 2 01:38:12.390483982 (658h9m12.261567027s) [upload=576561]: Done starting upload Jan 2 01:38:12.390523227 (658h9m12.261606292s) [upload=576561]: Uploading segments... Jan 2 01:38:12.390793081 (658h9m12.261876136s) [upload=576561]: Got next segment. Inline: false Jan 2 01:38:12.390832666 (658h9m12.261915711s) [upload=576561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:38:12.390838527 (658h9m12.261921582s) [upload=576561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:38:12.642804033 (658h9m12.513887068s) [seg_pos={0 0}, upload=576561]: Begin upload segment... Jan 2 01:38:12.642863557 (658h9m12.513946602s) [seg_pos={0 0}, upload=576561]: Done begin upload segment. Jan 2 01:38:12.642892541 (658h9m12.513975576s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.642902250 (658h9m12.513985295s) [noise=true , address=145.79.227.46:4602 , piece_id=4M4GWXMLUKKH7NV6Q23KZGKLZAUPOJYMFGJL2IDWFQWDBJ2GG45Q, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.643445063 (658h9m12.514528098s) [noise=true , address=145.79.227.50:4375 , piece_id=SDTNWJQ3PWKHFJYQZ5TQPR3U4L2KXLK5RLMGNAJMACJCI2ZA2XEQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.643876686 (658h9m12.514959721s) [noise=true , address=145.79.227.14:4838 , piece_id=PJV3IGOVX4BNBQQ7FXZWU7AHA4TBSDQOW5EYNAIER45ONTG3GI4Q, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.644285343 (658h9m12.515368388s) [noise=true , address=45.91.116.12:4227 , piece_id=4G7QXQDWOJJQOK22PNEWAP4HHVOV7T5FVHHXGUC3LUVQOFP6PLRA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.644699863 (658h9m12.515782898s) [noise=true , address=77.90.31.15:4700 , piece_id=WQX2OA6LHZREKK7N2UR3JXPUPVLR3OUAPMQLLV5BAO3EALFTZFHQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645008901 (658h9m12.516091946s) [noise=true , address=145.79.227.38:4978 , piece_id=O7LPFKZSOEZSQAJGD2GISKXW6DCQTVRHH4NL2PZJE2IKP7ZP5WQQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645338007 (658h9m12.516421042s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.645732098 (658h9m12.516815144s) [noise=true , address=145.79.227.18:4006 , piece_id=7OULP7JYSHUH7VMKBSLXSHL67Y4SFLXTCG44MVIAD3OSP5D6SDBA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.647470698 (658h9m12.518553743s) [noise=true , address=145.79.227.10:4401 , piece_id=CWO7NJO5KRU5AA3BLVGHLKV5CFNLQLZOJBQV3GYZDLKDSJ77Y7HA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.647639169 (658h9m12.518722214s) [noise=true , address=45.91.116.13:4153 , piece_id=BRAGSIR4JZYNLV3VVH2GS6IQQHTYJWNOVOLP5FX3GUXAU3SQGOZA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.649109919 (658h9m12.520192964s) [noise=true , address=77.90.31.10:4517 , piece_id=QK6A5K4H3ALLRWYJ42YPQSYDYTGNM6QLI4DBZGVTCKMBR6BSELZA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.649461709 (658h9m12.520544754s) [noise=true , address=145.79.227.42:4122 , piece_id=FAZUQRDBMJ4GMW5VFXR7UF37UCSYFOVFGAGYB76FYU46EJE7A7YQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.651511643 (658h9m12.522594688s) [noise=true , address=145.79.227.4:4187 , piece_id=R76LWRLP3ANDMWLOADX4IUBVWA4JVXX7N2XLYBHQDP6KD4VP2VIA, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.652031503 (658h9m12.523114548s) [noise=true , address=145.79.227.11:4732 , piece_id=SVVNDZB32RH7I43E3DSL2IQQ3QTW5DVINOHC6VDP64MSHEOG2QTQ, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:12.653941179 (658h9m12.525024234s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Uploading piece... Jan 2 01:38:14.347536041 (658h9m14.218619076s) [upload=576561]: Next returned nil segment Jan 2 01:38:14.347553323 (658h9m14.218636369s) [upload=576561]: Waiting for error group managing segments... Jan 2 01:38:14.626656891 (658h9m14.497739926s) [noise=true , address=145.79.227.46:4602 , piece_id=4M4GWXMLUKKH7NV6Q23KZGKLZAUPOJYMFGJL2IDWFQWDBJ2GG45Q, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.639934913 (658h9m14.511017958s) [noise=true , address=145.79.227.18:4006 , piece_id=7OULP7JYSHUH7VMKBSLXSHL67Y4SFLXTCG44MVIAD3OSP5D6SDBA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.640164981 (658h9m14.511248016s) [noise=true , address=145.79.227.50:4375 , piece_id=SDTNWJQ3PWKHFJYQZ5TQPR3U4L2KXLK5RLMGNAJMACJCI2ZA2XEQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.640697506 (658h9m14.511780541s) [noise=true , address=145.79.227.10:4401 , piece_id=CWO7NJO5KRU5AA3BLVGHLKV5CFNLQLZOJBQV3GYZDLKDSJ77Y7HA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.641188240 (658h9m14.512271285s) [noise=true , address=145.79.227.14:4838 , piece_id=PJV3IGOVX4BNBQQ7FXZWU7AHA4TBSDQOW5EYNAIER45ONTG3GI4Q, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.649071289 (658h9m14.520154324s) [noise=true , address=145.79.227.4:4187 , piece_id=R76LWRLP3ANDMWLOADX4IUBVWA4JVXX7N2XLYBHQDP6KD4VP2VIA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.663599412 (658h9m14.534682457s) [noise=true , address=145.79.227.42:4122 , piece_id=FAZUQRDBMJ4GMW5VFXR7UF37UCSYFOVFGAGYB76FYU46EJE7A7YQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.697036497 (658h9m14.568119542s) [noise=true , address=145.79.227.38:4978 , piece_id=O7LPFKZSOEZSQAJGD2GISKXW6DCQTVRHH4NL2PZJE2IKP7ZP5WQQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.728208401 (658h9m14.599291446s) [noise=true , address=145.79.227.11:4732 , piece_id=SVVNDZB32RH7I43E3DSL2IQQ3QTW5DVINOHC6VDP64MSHEOG2QTQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.893700516 (658h9m14.764783561s) [noise=true , address=77.90.31.10:4517 , piece_id=QK6A5K4H3ALLRWYJ42YPQSYDYTGNM6QLI4DBZGVTCKMBR6BSELZA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.912641551 (658h9m14.783724596s) [noise=true , address=77.90.31.15:4700 , piece_id=WQX2OA6LHZREKK7N2UR3JXPUPVLR3OUAPMQLLV5BAO3EALFTZFHQ, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.942977492 (658h9m14.814060527s) [noise=true , address=45.91.116.12:4227 , piece_id=4G7QXQDWOJJQOK22PNEWAP4HHVOV7T5FVHHXGUC3LUVQOFP6PLRA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.966907175 (658h9m14.83799021s) [noise=true , address=45.91.116.13:4153 , piece_id=BRAGSIR4JZYNLV3VVH2GS6IQQHTYJWNOVOLP5FX3GUXAU3SQGOZA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: <nil> Jan 2 01:38:14.966926713 (658h9m14.838009758s) [seg_pos={0 0}, upload=576561]: Segment reached optimal threshold of 13 pieces. Jan 2 01:38:14.967106485 (658h9m14.83818953s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 2 01:38:14.967113628 (658h9m14.838196663s) [noise=true , address=45.91.116.6:4976 , piece_id=H3AXNFK52QASJ4CBXWKIIRDX76QPISRXOHHXXX5DBXXACNNUXR3A, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967290826 (658h9m14.838373871s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:35442->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 2 01:38:14.967301647 (658h9m14.838384682s) [noise=true , address=45.91.116.4:4940 , piece_id=FFS66S6ZAXX3RPWKGRMTUDO4LPQGJVGFL5LOROR5W52DWTBTN3FA, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967346632 (658h9m14.838429667s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 2 01:38:14.967356381 (658h9m14.838439436s) [noise=true , address=45.91.116.5:4297 , piece_id=QB32DUA5HNSITSI7HQ3EZZ6JY2EJAMQGXNPNXOVTQGKKMOXYIRSA, seg_pos={0 0}, upload=576561]: Long tail cancellation due to optimal threshold Jan 2 01:38:14.967402248 (658h9m14.838485293s) [seg_pos={0 0}, upload=576561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:38:14.967479115 (658h9m14.83856215s) [upload=576561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:38:14.967485477 (658h9m14.838568522s) [upload=576561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:38:15.263533106 (658h9m15.134616141s) [upload=576561]: Done uploading segments... Jan 2 01:38:15.263557072 (658h9m15.134640107s) [upload=576561]: Upload finished. err: <nil> Duration: 30.003410092s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 165.968774ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:38:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 165.994852ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......XXX] duration: 30.002404786s trace: https://jaeger.dev.storj.io/trace/3c0b1ed35b6bdbcc FAIL (2026-01-02 01:37:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:37:19.933149274 (658h8m19.804232309s) [upload=576548]: Starting upload Jan 2 01:37:19.933224898 (658h8m19.804307943s) [upload=576548]: Done starting upload Jan 2 01:37:19.933590855 (658h8m19.80467389s) [upload=576548]: Uploading segments... Jan 2 01:37:19.933738576 (658h8m19.804821621s) [upload=576548]: Got next segment. Inline: false Jan 2 01:37:19.933779734 (658h8m19.804862779s) [upload=576548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:37:19.933785275 (658h8m19.80486831s) [upload=576548]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:37:20.182416972 (658h8m20.053500007s) [seg_pos={0 0}, upload=576548]: Begin upload segment... Jan 2 01:37:20.182464323 (658h8m20.053547358s) [seg_pos={0 0}, upload=576548]: Done begin upload segment. Jan 2 01:37:20.182494480 (658h8m20.053577515s) [noise=true , address=145.79.227.6:4259 , piece_id=D2FEYPENDT3VHRXVGNISKPBKJUBO6ZREBIRELBFPINOJDCBTWN3A, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.182645568 (658h8m20.053728603s) [noise=true , address=77.90.31.11:4554 , piece_id=BIIY3OD3H2DODTOYOSOIY6W72J7MRJS55HHYUMEUYUWCT4KZOJ5Q, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.183318308 (658h8m20.054401343s) [noise=true , address=145.79.227.4:4186 , piece_id=BOXKND755VIXVIDBF36JLCUVDJH6ZD7XF7WJIDNA3ZP24OOUTQ4A, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.183682753 (658h8m20.054765788s) [noise=true , address=77.90.31.14:4661 , piece_id=UFQ75L5AIHDSFZ3AN5ERRBPUBCYZVAKVUNWGL4LMUZSCUZ5YXUWA, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.184062055 (658h8m20.0551451s) [noise=true , address=45.91.116.6:4973 , piece_id=32R5RJHRIHYEESFZU2GI6PZJPX6NZVMXPC6DXCE6XTS2Z4SY6SNA, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.184235374 (658h8m20.055318409s) [noise=true , address=77.90.31.19:4842 , piece_id=3Y7CUXN2LAJHY3CVIPEO337ZSGVBMAQVEIA3RSWLKJQLD23C6VUA, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.184445816 (658h8m20.055528851s) [noise=true , address=45.91.116.4:4937 , piece_id=GFN7RNLSRFBUQPV76MLRDLMUHVUHJMG2CGLZLBBVVBFCOCI4B6RA, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.184580371 (658h8m20.055663416s) [noise=true , address=145.79.227.48:4302 , piece_id=VP5DDAIDGUSUPLDTNIVKBJDKNM4KAQ77YLYRWT5C4TK4AD2WRMRA, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.184814978 (658h8m20.055898023s) [noise=true , address=145.79.227.23:4545 , piece_id=CALKSOSRNZ5Q7Z27RXN7E5T6TFESF757LUTJ5G7URVU3TLNZQYAQ, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.185060726 (658h8m20.056143761s) [noise=true , address=77.90.31.15:4700 , piece_id=WPFLMRTKX5GA273QUCE6TIWMH7Q5BA7UFGSI7D2VLJDIZJKLUNJA, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.185195884 (658h8m20.056278919s) [noise=true , address=45.91.116.13:4155 , piece_id=ZERCPMTCJNYSS7SXHG2QYZJFSZYH6TIGCDOSDFXY4ZYGUSXP7MDQ, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.185481688 (658h8m20.056564723s) [noise=true , address=145.79.227.38:4977 , piece_id=ZBYTM3JZ73RQI75XGL576JKGA5Z5MOEGUZRAR5VBN74ZCTJIHRNA, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.185565027 (658h8m20.056648062s) [noise=true , address=145.79.227.36:4518 , piece_id=AIOGH6ALYSUXDKBZ6AGAYLHHU6B5JF3SWAVIRJS2YUEK3MKSIBQQ, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.185884034 (658h8m20.056967069s) [noise=true , address=145.79.227.9:4365 , piece_id=444KA5E4QJQHDKOLAI24AN3NXO3AEW5ODD5UFZCZIVROF54QLGQQ, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.186005345 (658h8m20.05708839s) [noise=true , address=145.79.227.13:4803 , piece_id=J3PYBNFZHWCQ4DDUTZ5JKZGDSRMTDT6SXQE56HI3FNTP7X2L6XJQ, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:20.186395998 (658h8m20.057479044s) [noise=true , address=45.91.116.12:4225 , piece_id=VYVVNCA2MZKHTEVW3BKZM35OT7VACDXYGNQ6YIFSBONQEUPOCSWA, seg_pos={0 0}, upload=576548]: Uploading piece... Jan 2 01:37:21.857041234 (658h8m21.728124269s) [upload=576548]: Next returned nil segment Jan 2 01:37:21.857052996 (658h8m21.728136031s) [upload=576548]: Waiting for error group managing segments... Jan 2 01:37:22.160995527 (658h8m22.032078562s) [noise=true , address=145.79.227.36:4518 , piece_id=AIOGH6ALYSUXDKBZ6AGAYLHHU6B5JF3SWAVIRJS2YUEK3MKSIBQQ, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.161404957 (658h8m22.032487992s) [noise=true , address=145.79.227.23:4545 , piece_id=CALKSOSRNZ5Q7Z27RXN7E5T6TFESF757LUTJ5G7URVU3TLNZQYAQ, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.162952464 (658h8m22.034035499s) [noise=true , address=145.79.227.48:4302 , piece_id=VP5DDAIDGUSUPLDTNIVKBJDKNM4KAQ77YLYRWT5C4TK4AD2WRMRA, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.199196584 (658h8m22.07027963s) [noise=true , address=145.79.227.4:4186 , piece_id=BOXKND755VIXVIDBF36JLCUVDJH6ZD7XF7WJIDNA3ZP24OOUTQ4A, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.199799934 (658h8m22.070882979s) [noise=true , address=145.79.227.13:4803 , piece_id=J3PYBNFZHWCQ4DDUTZ5JKZGDSRMTDT6SXQE56HI3FNTP7X2L6XJQ, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.206267809 (658h8m22.077350844s) [noise=true , address=145.79.227.9:4365 , piece_id=444KA5E4QJQHDKOLAI24AN3NXO3AEW5ODD5UFZCZIVROF54QLGQQ, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.210686462 (658h8m22.081769497s) [noise=true , address=145.79.227.6:4259 , piece_id=D2FEYPENDT3VHRXVGNISKPBKJUBO6ZREBIRELBFPINOJDCBTWN3A, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.264428060 (658h8m22.135511105s) [noise=true , address=145.79.227.38:4977 , piece_id=ZBYTM3JZ73RQI75XGL576JKGA5Z5MOEGUZRAR5VBN74ZCTJIHRNA, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.368886864 (658h8m22.239969899s) [noise=true , address=77.90.31.15:4700 , piece_id=WPFLMRTKX5GA273QUCE6TIWMH7Q5BA7UFGSI7D2VLJDIZJKLUNJA, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.442642588 (658h8m22.313725633s) [noise=true , address=77.90.31.14:4661 , piece_id=UFQ75L5AIHDSFZ3AN5ERRBPUBCYZVAKVUNWGL4LMUZSCUZ5YXUWA, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.450768150 (658h8m22.321851205s) [noise=true , address=77.90.31.11:4554 , piece_id=BIIY3OD3H2DODTOYOSOIY6W72J7MRJS55HHYUMEUYUWCT4KZOJ5Q, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.491410626 (658h8m22.362493661s) [noise=true , address=45.91.116.4:4937 , piece_id=GFN7RNLSRFBUQPV76MLRDLMUHVUHJMG2CGLZLBBVVBFCOCI4B6RA, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.600564811 (658h8m22.471647856s) [noise=true , address=45.91.116.6:4973 , piece_id=32R5RJHRIHYEESFZU2GI6PZJPX6NZVMXPC6DXCE6XTS2Z4SY6SNA, seg_pos={0 0}, upload=576548]: Done uploading piece. err: <nil> Jan 2 01:37:22.600587283 (658h8m22.471670318s) [seg_pos={0 0}, upload=576548]: Segment reached optimal threshold of 13 pieces. Jan 2 01:37:22.600918554 (658h8m22.472001589s) [noise=true , address=45.91.116.13:4155 , piece_id=ZERCPMTCJNYSS7SXHG2QYZJFSZYH6TIGCDOSDFXY4ZYGUSXP7MDQ, seg_pos={0 0}, upload=576548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jan 2 01:37:22.600927932 (658h8m22.472010967s) [noise=true , address=45.91.116.13:4155 , piece_id=ZERCPMTCJNYSS7SXHG2QYZJFSZYH6TIGCDOSDFXY4ZYGUSXP7MDQ, seg_pos={0 0}, upload=576548]: Long tail cancellation due to optimal threshold Jan 2 01:37:22.601004027 (658h8m22.472087062s) [noise=true , address=77.90.31.19:4842 , piece_id=3Y7CUXN2LAJHY3CVIPEO337ZSGVBMAQVEIA3RSWLKJQLD23C6VUA, seg_pos={0 0}, upload=576548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jan 2 01:37:22.601023985 (658h8m22.47210702s) [noise=true , address=77.90.31.19:4842 , piece_id=3Y7CUXN2LAJHY3CVIPEO337ZSGVBMAQVEIA3RSWLKJQLD23C6VUA, seg_pos={0 0}, upload=576548]: Long tail cancellation due to optimal threshold Jan 2 01:37:22.601163992 (658h8m22.472247027s) [noise=true , address=45.91.116.12:4225 , piece_id=VYVVNCA2MZKHTEVW3BKZM35OT7VACDXYGNQ6YIFSBONQEUPOCSWA, seg_pos={0 0}, upload=576548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jan 2 01:37:22.601171466 (658h8m22.472254511s) [noise=true , address=45.91.116.12:4225 , piece_id=VYVVNCA2MZKHTEVW3BKZM35OT7VACDXYGNQ6YIFSBONQEUPOCSWA, seg_pos={0 0}, upload=576548]: Long tail cancellation due to optimal threshold Jan 2 01:37:22.601192386 (658h8m22.472275421s) [seg_pos={0 0}, upload=576548]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:37:22.601320690 (658h8m22.472403725s) [upload=576548]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:37:22.601336180 (658h8m22.472419215s) [upload=576548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:37:22.891686249 (658h8m22.762769305s) [upload=576548]: Done uploading segments... Jan 2 01:37:22.891792822 (658h8m22.762875857s) [upload=576548]: Upload finished. err: <nil> Duration: 30.00242837s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.344934116s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:37:44 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.344963212s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.286139379s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:37:12 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.286169547s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 185.57734ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:37:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 185.616424ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......XX] duration: 30.002561451s trace: https://jaeger.dev.storj.io/trace/3ef0f2646c618d5 FAIL (2026-01-02 01:36:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7R7S47KLQJ75X2GA4Z76EIQZGO4GGMRZAPVXUAC43ZPTMYTPT3ZQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: LN3JCAB3BROD622SQMJH4UWYAMX2NJUPRBROT4COEOVJ4UKRTMKQ) context deadline exceeded ======LOGS====== 58h7m19.096855165s) [upload=576532]: Uploading segments... Jan 2 01:36:19.225903962 (658h7m19.096987007s) [upload=576532]: Got next segment. Inline: false Jan 2 01:36:19.225944579 (658h7m19.097027624s) [upload=576532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:36:19.225991718 (658h7m19.097074773s) [upload=576532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:36:19.472961712 (658h7m19.344044747s) [seg_pos={0 0}, upload=576532]: Begin upload segment... Jan 2 01:36:19.473040582 (658h7m19.344123617s) [seg_pos={0 0}, upload=576532]: Done begin upload segment. Jan 2 01:36:19.473072753 (658h7m19.344155798s) [noise=true , address=145.79.227.41:4087 , piece_id=WY4OZBPCE2QGBJD73IAQK4PNSYHMHJQ6SIIKBK5T2L7UGOC2QR5Q, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.473142456 (658h7m19.344225491s) [noise=true , address=77.90.31.13:4626 , piece_id=LN3JCAB3BROD622SQMJH4UWYAMX2NJUPRBROT4COEOVJ4UKRTMKQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.473888737 (658h7m19.344971772s) [noise=true , address=145.79.227.3:4149 , piece_id=7CENKO6VFBZWK7VSE5AC4S5G5PXCHAANRK2H6J5BWPHW7YMKNCAQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.473910769 (658h7m19.344993804s) [noise=true , address=145.79.227.14:4840 , piece_id=FKDYIUHII5GIWUPOVQCJTB32LSXGBQZXI2XERECXCYUZNK35IJKQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.474398428 (658h7m19.345481463s) [noise=true , address=45.91.116.5:4299 , piece_id=7R7S47KLQJ75X2GA4Z76EIQZGO4GGMRZAPVXUAC43ZPTMYTPT3ZQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.474450206 (658h7m19.345533251s) [noise=true , address=77.90.31.15:4700 , piece_id=GDABZJABJU3KAHIRILC3JYMZQKP22FAN4ZLMGCBZXLUV7KXVI6JA, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.474749757 (658h7m19.345832792s) [noise=true , address=145.79.227.6:4259 , piece_id=IM72MAH7YL3APWEPO2NYGNDCLK2QTYQIFMG5KG5UEPR6SCMYFDWQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.474954456 (658h7m19.346037492s) [noise=true , address=145.79.227.25:4618 , piece_id=RB42PNRGBORLAMBPERI3TKN2YTYFLY43GQDIOM22PKZ2EJSNLHOQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.475137885 (658h7m19.346220931s) [noise=true , address=77.90.31.21:4878 , piece_id=X2NPXM7FSK7V6WF6YGRYBE5JP2RT4LDOSUAD25U453VE543FPOBQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.475491939 (658h7m19.346574975s) [noise=true , address=145.79.227.47:4267 , piece_id=VLU6STG4HMP2RU2KXNUW7U4EDBXPIHCUQ4WDQTADSY3Y4VZCY4DQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.476180481 (658h7m19.347263526s) [noise=true , address=145.79.227.46:4603 , piece_id=ZJUCDFM4BERU6FDSNZRAYCL733YSZEXJGG2224J2M4WIY6ACUIQQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.478087112 (658h7m19.349170147s) [noise=true , address=145.79.227.45:4231 , piece_id=2POJGO3GEKT55YRX2CJ477ULXXBZJ2R5G7TAVBBWHCWECQC2QFBA, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.478528041 (658h7m19.349611076s) [noise=true , address=145.79.227.39:4013 , piece_id=4L7IDOYUR334A2G7EB57PGCOJTYXD33SBENDSEQKBB2VF5D6MMVA, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.479130097 (658h7m19.350213132s) [noise=true , address=45.91.116.6:4975 , piece_id=GNINSHVSRKGDY3P6FYGGE64UIMOWQVZF5BQDODCORB2ZWOYITU6Q, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.483305818 (658h7m19.354388853s) [noise=true , address=145.79.227.31:4797 , piece_id=5QT2EB2TRNC54N7XTWPRWH5CJIMAXCPJU3R2MINXRKRRXJHFHSGA, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:19.488545966 (658h7m19.359629001s) [noise=true , address=77.90.31.17:4773 , piece_id=SPCSQ2TOLQFFO37YXLQ5Y2GJH5SEYF2JBD7JUFEIZJRVHPR5SSBQ, seg_pos={0 0}, upload=576532]: Uploading piece... Jan 2 01:36:21.170573052 (658h7m21.041656097s) [upload=576532]: Next returned nil segment Jan 2 01:36:21.170584343 (658h7m21.041667378s) [upload=576532]: Waiting for error group managing segments... Jan 2 01:36:21.450672476 (658h7m21.321755541s) [noise=true , address=145.79.227.3:4149 , piece_id=7CENKO6VFBZWK7VSE5AC4S5G5PXCHAANRK2H6J5BWPHW7YMKNCAQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.464752454 (658h7m21.335835489s) [noise=true , address=145.79.227.46:4603 , piece_id=ZJUCDFM4BERU6FDSNZRAYCL733YSZEXJGG2224J2M4WIY6ACUIQQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.468156085 (658h7m21.33923913s) [noise=true , address=145.79.227.14:4840 , piece_id=FKDYIUHII5GIWUPOVQCJTB32LSXGBQZXI2XERECXCYUZNK35IJKQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.468520359 (658h7m21.339603394s) [noise=true , address=145.79.227.25:4618 , piece_id=RB42PNRGBORLAMBPERI3TKN2YTYFLY43GQDIOM22PKZ2EJSNLHOQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.476828387 (658h7m21.347911422s) [noise=true , address=145.79.227.47:4267 , piece_id=VLU6STG4HMP2RU2KXNUW7U4EDBXPIHCUQ4WDQTADSY3Y4VZCY4DQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.484464858 (658h7m21.355547903s) [noise=true , address=145.79.227.41:4087 , piece_id=WY4OZBPCE2QGBJD73IAQK4PNSYHMHJQ6SIIKBK5T2L7UGOC2QR5Q, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.485857279 (658h7m21.356940334s) [noise=true , address=145.79.227.6:4259 , piece_id=IM72MAH7YL3APWEPO2NYGNDCLK2QTYQIFMG5KG5UEPR6SCMYFDWQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.489704154 (658h7m21.360787189s) [noise=true , address=145.79.227.45:4231 , piece_id=2POJGO3GEKT55YRX2CJ477ULXXBZJ2R5G7TAVBBWHCWECQC2QFBA, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.497882336 (658h7m21.368965381s) [noise=true , address=145.79.227.31:4797 , piece_id=5QT2EB2TRNC54N7XTWPRWH5CJIMAXCPJU3R2MINXRKRRXJHFHSGA, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.563384956 (658h7m21.434468001s) [noise=true , address=145.79.227.39:4013 , piece_id=4L7IDOYUR334A2G7EB57PGCOJTYXD33SBENDSEQKBB2VF5D6MMVA, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.661940791 (658h7m21.533023837s) [noise=true , address=77.90.31.15:4700 , piece_id=GDABZJABJU3KAHIRILC3JYMZQKP22FAN4ZLMGCBZXLUV7KXVI6JA, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.754602123 (658h7m21.625685158s) [noise=true , address=77.90.31.13:4626 , piece_id=LN3JCAB3BROD622SQMJH4UWYAMX2NJUPRBROT4COEOVJ4UKRTMKQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.934702624 (658h7m21.805785659s) [noise=true , address=45.91.116.5:4299 , piece_id=7R7S47KLQJ75X2GA4Z76EIQZGO4GGMRZAPVXUAC43ZPTMYTPT3ZQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: <nil> Jan 2 01:36:21.934720969 (658h7m21.805804014s) [seg_pos={0 0}, upload=576532]: Segment reached optimal threshold of 13 pieces. Jan 2 01:36:21.934898136 (658h7m21.805981171s) [noise=true , address=77.90.31.17:4773 , piece_id=SPCSQ2TOLQFFO37YXLQ5Y2GJH5SEYF2JBD7JUFEIZJRVHPR5SSBQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:33752->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:33752->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33752->77.90.31.17:4773: use of closed network connection Jan 2 01:36:21.934903606 (658h7m21.805986642s) [noise=true , address=77.90.31.17:4773 , piece_id=SPCSQ2TOLQFFO37YXLQ5Y2GJH5SEYF2JBD7JUFEIZJRVHPR5SSBQ, seg_pos={0 0}, upload=576532]: Long tail cancellation due to optimal threshold Jan 2 01:36:21.935062349 (658h7m21.806145404s) [noise=true , address=45.91.116.6:4975 , piece_id=GNINSHVSRKGDY3P6FYGGE64UIMOWQVZF5BQDODCORB2ZWOYITU6Q, seg_pos={0 0}, upload=576532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Jan 2 01:36:21.935081685 (658h7m21.806164721s) [noise=true , address=45.91.116.6:4975 , piece_id=GNINSHVSRKGDY3P6FYGGE64UIMOWQVZF5BQDODCORB2ZWOYITU6Q, seg_pos={0 0}, upload=576532]: Long tail cancellation due to optimal threshold Jan 2 01:36:21.935103277 (658h7m21.806186312s) [noise=true , address=77.90.31.21:4878 , piece_id=X2NPXM7FSK7V6WF6YGRYBE5JP2RT4LDOSUAD25U453VE543FPOBQ, seg_pos={0 0}, upload=576532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:51542->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:51542->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51542->77.90.31.21:4878: use of closed network connection Jan 2 01:36:21.935111753 (658h7m21.806194788s) [noise=true , address=77.90.31.21:4878 , piece_id=X2NPXM7FSK7V6WF6YGRYBE5JP2RT4LDOSUAD25U453VE543FPOBQ, seg_pos={0 0}, upload=576532]: Long tail cancellation due to optimal threshold Jan 2 01:36:21.935212545 (658h7m21.80629558s) [seg_pos={0 0}, upload=576532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:36:21.935303799 (658h7m21.806386834s) [upload=576532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:36:21.935309299 (658h7m21.806392334s) [upload=576532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:36:22.234895065 (658h7m22.10597811s) [upload=576532]: Done uploading segments... Jan 2 01:36:22.234914493 (658h7m22.105997528s) [upload=576532]: Upload finished. err: <nil> Duration: 30.002579536s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.151908887s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:36:26 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.151937873s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 205.451921ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:36:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 205.482128ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......X] duration: 30.003086505s trace: https://jaeger.dev.storj.io/trace/7c9c8cc7d8db45ef FAIL (2026-01-02 01:35:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: F7WIKEXOK4A6XC73O2R6RW6FMU5C65WDFBLOHE5NVOX4PEW3PC3Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: CAPKI4XMMBIU3MPXX43DV7H7SBWNVZOWU5SI5SU24WKOWGUGW34Q) context deadline exceeded ======LOGS====== Jan 2 01:35:20.573321188 (658h6m20.444404223s) [upload=576517]: Starting upload Jan 2 01:35:20.573411821 (658h6m20.444494866s) [upload=576517]: Done starting upload Jan 2 01:35:20.573509518 (658h6m20.444592573s) [upload=576517]: Uploading segments... Jan 2 01:35:20.573625799 (658h6m20.444708844s) [upload=576517]: Got next segment. Inline: false Jan 2 01:35:20.573678209 (658h6m20.444761254s) [upload=576517]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:35:20.573686524 (658h6m20.44476958s) [upload=576517]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:35:20.816001695 (658h6m20.68708473s) [seg_pos={0 0}, upload=576517]: Begin upload segment... Jan 2 01:35:20.816052442 (658h6m20.687135477s) [seg_pos={0 0}, upload=576517]: Done begin upload segment. Jan 2 01:35:20.816080175 (658h6m20.68716321s) [noise=true , address=145.79.227.30:4762 , piece_id=XVBEDRKUFVS7ZWYLG6M4NFSKN6MAMATFOTFWZR7FHJJWGNSANNRA, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.816221584 (658h6m20.687304649s) [noise=true , address=145.79.227.4:4185 , piece_id=AQXC3S4OBWBGIVI6T3PVRPVTUIBZTEST6RRJH64CWV6AJV4Z4SRQ, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.816839480 (658h6m20.687922515s) [noise=true , address=145.79.227.36:4517 , piece_id=2EXOM3KKARTVSPXBEJ5GJIEEALELBDK45RGMBRTDOLJ7OYBHGGDQ, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.817264319 (658h6m20.688347354s) [noise=true , address=45.91.116.5:4298 , piece_id=F7WIKEXOK4A6XC73O2R6RW6FMU5C65WDFBLOHE5NVOX4PEW3PC3Q, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.817730106 (658h6m20.688813151s) [noise=true , address=145.79.227.50:4373 , piece_id=H6DYRUI5EUVUXFGJ5MMURQAR3MRF3PWMSJWNCLO5HWBK4KBME23A, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.817999488 (658h6m20.689082523s) [noise=true , address=45.91.116.6:4973 , piece_id=SNKJR6R5UXB5CD4QAOTD2DXDUSND3SBKNCY5KFYT4ORVNH6Q4P6A, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.818209428 (658h6m20.689292463s) [noise=true , address=45.91.116.9:4263 , piece_id=EYOBMNSNYACPZH5MXVGI44UHQNPJO6BGA63UUE7ERCVT4M5D3U4Q, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.818428736 (658h6m20.689511781s) [noise=true , address=145.79.227.46:4602 , piece_id=APAZNFQLEFKQPEZ5JFWMATCVTAZFLGZGL4L5QKS26ZKYGDCLMLQA, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.818514469 (658h6m20.689597514s) [noise=true , address=145.79.227.6:4258 , piece_id=Y2LXZQTG6SURSPFOLXEZFIHDSL7AB4KA7N4RONL6OZ4ENOYNKH3A, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.818819610 (658h6m20.689902655s) [noise=true , address=145.79.227.13:4804 , piece_id=ALCIG4TYH7I7B3O6V5E2T5FR46YNFD4H4RSUM5AENFRMURMIUCDQ, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.818876940 (658h6m20.689959985s) [noise=true , address=145.79.227.48:4302 , piece_id=ZCFGID7KALXZD2DJQGQRJHMYJ4JP3IXYGV33I5WYV2S6AMYGQONA, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.819142926 (658h6m20.690225961s) [noise=true , address=145.79.227.35:4481 , piece_id=P6AI7Y57GXZEE5A2H4VYER4JVY7C7TKGSQZIGPTYUQ356ADVAJUQ, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.819337356 (658h6m20.690420401s) [noise=true , address=145.79.227.38:4977 , piece_id=ZEPUJNZWWSZY54XLFULEGGV4JLX6MCLVKI2SXCAG5QSGLDK3BESQ, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.819442927 (658h6m20.690525962s) [noise=true , address=145.79.227.34:4905 , piece_id=X3IBIBFAOPZ2QOPWWOKOMTIOJFQPF5LJKJ5UGM562OF7ZDPUYFQQ, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.819734272 (658h6m20.690817317s) [noise=true , address=145.79.227.12:4767 , piece_id=XJA34JNFLXOPVY7S7N5VANW2P2OKPAYE755GBBPCBLA3YUFL6MVQ, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:20.819795589 (658h6m20.690878634s) [noise=true , address=45.91.116.8:4045 , piece_id=CAPKI4XMMBIU3MPXX43DV7H7SBWNVZOWU5SI5SU24WKOWGUGW34Q, seg_pos={0 0}, upload=576517]: Uploading piece... Jan 2 01:35:22.428085185 (658h6m22.29916823s) [upload=576517]: Next returned nil segment Jan 2 01:35:22.428109592 (658h6m22.299192637s) [upload=576517]: Waiting for error group managing segments... Jan 2 01:35:22.809122881 (658h6m22.680205916s) [noise=true , address=145.79.227.48:4302 , piece_id=ZCFGID7KALXZD2DJQGQRJHMYJ4JP3IXYGV33I5WYV2S6AMYGQONA, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:22.809290701 (658h6m22.680373746s) [noise=true , address=145.79.227.4:4185 , piece_id=AQXC3S4OBWBGIVI6T3PVRPVTUIBZTEST6RRJH64CWV6AJV4Z4SRQ, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:22.810552142 (658h6m22.681635177s) [noise=true , address=145.79.227.12:4767 , piece_id=XJA34JNFLXOPVY7S7N5VANW2P2OKPAYE755GBBPCBLA3YUFL6MVQ, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:22.820013757 (658h6m22.691096802s) [noise=true , address=145.79.227.46:4602 , piece_id=APAZNFQLEFKQPEZ5JFWMATCVTAZFLGZGL4L5QKS26ZKYGDCLMLQA, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:22.820504622 (658h6m22.691587657s) [noise=true , address=145.79.227.6:4258 , piece_id=Y2LXZQTG6SURSPFOLXEZFIHDSL7AB4KA7N4RONL6OZ4ENOYNKH3A, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:22.826966816 (658h6m22.698049851s) [noise=true , address=145.79.227.38:4977 , piece_id=ZEPUJNZWWSZY54XLFULEGGV4JLX6MCLVKI2SXCAG5QSGLDK3BESQ, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:22.872098890 (658h6m22.743181925s) [noise=true , address=145.79.227.36:4517 , piece_id=2EXOM3KKARTVSPXBEJ5GJIEEALELBDK45RGMBRTDOLJ7OYBHGGDQ, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:22.885919705 (658h6m22.75700275s) [noise=true , address=145.79.227.50:4373 , piece_id=H6DYRUI5EUVUXFGJ5MMURQAR3MRF3PWMSJWNCLO5HWBK4KBME23A, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:23.001624583 (658h6m22.872707629s) [noise=true , address=145.79.227.13:4804 , piece_id=ALCIG4TYH7I7B3O6V5E2T5FR46YNFD4H4RSUM5AENFRMURMIUCDQ, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:23.102650342 (658h6m22.973733377s) [noise=true , address=145.79.227.35:4481 , piece_id=P6AI7Y57GXZEE5A2H4VYER4JVY7C7TKGSQZIGPTYUQ356ADVAJUQ, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:23.136296004 (658h6m23.00737905s) [noise=true , address=45.91.116.5:4298 , piece_id=F7WIKEXOK4A6XC73O2R6RW6FMU5C65WDFBLOHE5NVOX4PEW3PC3Q, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:23.245114330 (658h6m23.116197365s) [noise=true , address=45.91.116.6:4973 , piece_id=SNKJR6R5UXB5CD4QAOTD2DXDUSND3SBKNCY5KFYT4ORVNH6Q4P6A, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:23.251391172 (658h6m23.122474217s) [noise=true , address=45.91.116.8:4045 , piece_id=CAPKI4XMMBIU3MPXX43DV7H7SBWNVZOWU5SI5SU24WKOWGUGW34Q, seg_pos={0 0}, upload=576517]: Done uploading piece. err: <nil> Jan 2 01:35:23.251416139 (658h6m23.122499184s) [seg_pos={0 0}, upload=576517]: Segment reached optimal threshold of 13 pieces. Jan 2 01:35:23.251648522 (658h6m23.122731587s) [noise=true , address=145.79.227.34:4905 , piece_id=X3IBIBFAOPZ2QOPWWOKOMTIOJFQPF5LJKJ5UGM562OF7ZDPUYFQQ, seg_pos={0 0}, upload=576517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 2 01:35:23.251658210 (658h6m23.122741245s) [noise=true , address=145.79.227.34:4905 , piece_id=X3IBIBFAOPZ2QOPWWOKOMTIOJFQPF5LJKJ5UGM562OF7ZDPUYFQQ, seg_pos={0 0}, upload=576517]: Long tail cancellation due to optimal threshold Jan 2 01:35:23.251856468 (658h6m23.122939523s) [noise=true , address=145.79.227.30:4762 , piece_id=XVBEDRKUFVS7ZWYLG6M4NFSKN6MAMATFOTFWZR7FHJJWGNSANNRA, seg_pos={0 0}, upload=576517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Jan 2 01:35:23.251867468 (658h6m23.122950504s) [noise=true , address=145.79.227.30:4762 , piece_id=XVBEDRKUFVS7ZWYLG6M4NFSKN6MAMATFOTFWZR7FHJJWGNSANNRA, seg_pos={0 0}, upload=576517]: Long tail cancellation due to optimal threshold Jan 2 01:35:23.251952160 (658h6m23.123035205s) [noise=true , address=45.91.116.9:4263 , piece_id=EYOBMNSNYACPZH5MXVGI44UHQNPJO6BGA63UUE7ERCVT4M5D3U4Q, seg_pos={0 0}, upload=576517]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:40624->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:40624->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40624->45.91.116.9:4263: use of closed network connection Jan 2 01:35:23.251962069 (658h6m23.123045114s) [noise=true , address=45.91.116.9:4263 , piece_id=EYOBMNSNYACPZH5MXVGI44UHQNPJO6BGA63UUE7ERCVT4M5D3U4Q, seg_pos={0 0}, upload=576517]: Long tail cancellation due to optimal threshold Jan 2 01:35:23.252003347 (658h6m23.123086382s) [seg_pos={0 0}, upload=576517]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:35:23.252094461 (658h6m23.123177506s) [upload=576517]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:35:23.252101695 (658h6m23.12318474s) [upload=576517]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:35:23.548376757 (658h6m23.419459792s) [upload=576517]: Done uploading segments... Jan 2 01:35:23.548399490 (658h6m23.419482535s) [upload=576517]: Upload finished. err: <nil> Duration: 30.003107896s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.490623856s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:35:24 UTC) orbiter warning: context deadline exceeded Duration: 3.490650145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.333600037s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:35:21 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.333630384s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 215.458545ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:35:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 215.47707ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X..X....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X......] duration: 22.164945498s trace: https://jaeger.dev.storj.io/trace/5f2ae0ac088f21fb FAIL (2026-01-02 01:34:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q) context deadline exceeded ======LOGS====== Jan 2 01:28:48.865298164 (657h59m48.736381199s) [upload=576429]: Starting upload Jan 2 01:28:48.865409055 (657h59m48.73649209s) [upload=576429]: Done starting upload Jan 2 01:28:48.865527081 (657h59m48.736610126s) [upload=576429]: Uploading segments... Jan 2 01:28:48.865619115 (657h59m48.73670215s) [upload=576429]: Got next segment. Inline: false Jan 2 01:28:48.865659492 (657h59m48.736742537s) [upload=576429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:28:48.865666045 (657h59m48.73674908s) [upload=576429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:28:49.108468076 (657h59m48.979551122s) [seg_pos={0 0}, upload=576429]: Begin upload segment... Jan 2 01:28:49.108512270 (657h59m48.979595305s) [seg_pos={0 0}, upload=576429]: Done begin upload segment. Jan 2 01:28:49.108557808 (657h59m48.979640863s) [noise=true , address=145.79.227.16:4912 , piece_id=PDGWWSRYIFHLH426P7FWIJHYQOVYVFVK2EQPAB4NCMBNEY2A632A, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.108583075 (657h59m48.97966612s) [noise=true , address=145.79.227.18:4007 , piece_id=WBCAAEZDZR4ROB4TX5GMS5INGZ5ZNYUM6D6T4VCE5GV2CXZLUC5A, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.109078840 (657h59m48.980161895s) [noise=true , address=145.79.227.37:4566 , piece_id=GAK64W3EPYBKQSJC6CZO3SSTS3XXUPDWSWALIACBZ4GPLIWNSS5Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.109498178 (657h59m48.980581233s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.110068083 (657h59m48.981151128s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.110529702 (657h59m48.981612747s) [noise=true , address=145.79.227.27:4690 , piece_id=JWZ2PKIOHGGB223CPLR77QWSQW5S3VVRKJKVF5UNJPTLZIYNCZLA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.111011129 (657h59m48.982094174s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.112367001 (657h59m48.983450036s) [noise=true , address=145.79.227.7:4294 , piece_id=W3BDYLEBEUII2FFWAD4ODWQ4YLF4IJJT3UF5JLZ23VUHHEIFWKWA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.112743067 (657h59m48.983826112s) [noise=true , address=45.91.116.5:4299 , piece_id=KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.113497813 (657h59m48.984580859s) [noise=true , address=77.90.31.17:4770 , piece_id=WQYNR72547D7ORNETLVURASVFQS237H67EXANZGZE4BWW4ERRVEQ, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.114878823 (657h59m48.985961869s) [noise=true , address=145.79.227.15:4876 , piece_id=KJLPSVOG4T6ABBYRQAC5ODG6U5OKQC6THJK7JGL7FQHZ62MDACWA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.116961329 (657h59m48.988044384s) [noise=true , address=145.79.227.52:4447 , piece_id=DYNXASJFMQ72IQ3KTPMEH2L5FPXXCXWQTCMIPZN3ZANAHNRA6DIA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.118513995 (657h59m48.98959704s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.120245753 (657h59m48.991328798s) [noise=true , address=145.79.227.48:4303 , piece_id=WQ7QVUF2CC2E23SQFFYTFDT7VPVHZFCAIP4P7A6EMR7CIQGMTLOA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.120462736 (657h59m48.991545781s) [noise=true , address=145.79.227.5:4221 , piece_id=4RU3WR7YTQD6W5I47ATFSHKLCYK3CX5DJOAX4VB7NQ24NSVLBSJA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.121557330 (657h59m48.992640365s) [noise=true , address=77.90.31.13:4626 , piece_id=UWAE4R62SWC2X6H56QZE7RS4OBEUWOLEXDCXFTLIZ2JFNHTXVKOQ, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:50.815693191 (657h59m50.686776226s) [upload=576429]: Next returned nil segment Jan 2 01:28:50.815711606 (657h59m50.686794642s) [upload=576429]: Waiting for error group managing segments... Jan 2 01:28:51.105308571 (657h59m50.976391616s) [noise=true , address=145.79.227.16:4912 , piece_id=PDGWWSRYIFHLH426P7FWIJHYQOVYVFVK2EQPAB4NCMBNEY2A632A, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.106445306 (657h59m50.977528341s) [noise=true , address=145.79.227.37:4566 , piece_id=GAK64W3EPYBKQSJC6CZO3SSTS3XXUPDWSWALIACBZ4GPLIWNSS5Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.106496463 (657h59m50.977579508s) [noise=true , address=145.79.227.5:4221 , piece_id=4RU3WR7YTQD6W5I47ATFSHKLCYK3CX5DJOAX4VB7NQ24NSVLBSJA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.114999614 (657h59m50.986082659s) [noise=true , address=145.79.227.52:4447 , piece_id=DYNXASJFMQ72IQ3KTPMEH2L5FPXXCXWQTCMIPZN3ZANAHNRA6DIA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.115736637 (657h59m50.986819682s) [noise=true , address=145.79.227.48:4303 , piece_id=WQ7QVUF2CC2E23SQFFYTFDT7VPVHZFCAIP4P7A6EMR7CIQGMTLOA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.116229075 (657h59m50.98731212s) [noise=true , address=145.79.227.27:4690 , piece_id=JWZ2PKIOHGGB223CPLR77QWSQW5S3VVRKJKVF5UNJPTLZIYNCZLA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.129635131 (657h59m51.000718176s) [noise=true , address=145.79.227.15:4876 , piece_id=KJLPSVOG4T6ABBYRQAC5ODG6U5OKQC6THJK7JGL7FQHZ62MDACWA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.136267359 (657h59m51.007350395s) [noise=true , address=145.79.227.7:4294 , piece_id=W3BDYLEBEUII2FFWAD4ODWQ4YLF4IJJT3UF5JLZ23VUHHEIFWKWA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.250622625 (657h59m51.12170566s) [noise=true , address=145.79.227.18:4007 , piece_id=WBCAAEZDZR4ROB4TX5GMS5INGZ5ZNYUM6D6T4VCE5GV2CXZLUC5A, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.364460585 (657h59m51.23554363s) [noise=true , address=77.90.31.13:4626 , piece_id=UWAE4R62SWC2X6H56QZE7RS4OBEUWOLEXDCXFTLIZ2JFNHTXVKOQ, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.376934517 (657h59m51.248017552s) [noise=true , address=77.90.31.17:4770 , piece_id=WQYNR72547D7ORNETLVURASVFQS237H67EXANZGZE4BWW4ERRVEQ, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.613301886 (657h59m51.484384931s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.633527909 (657h59m51.504610954s) [noise=true , address=45.91.116.5:4299 , piece_id=KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.633550943 (657h59m51.504633988s) [seg_pos={0 0}, upload=576429]: Segment reached optimal threshold of 13 pieces. Jan 2 01:28:51.633771082 (657h59m51.504854127s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection Jan 2 01:28:51.633781091 (657h59m51.504864126s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.633851505 (657h59m51.50493454s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Jan 2 01:28:51.633861725 (657h59m51.50494477s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.634178087 (657h59m51.505261132s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Jan 2 01:28:51.634187896 (657h59m51.505270931s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.634209056 (657h59m51.505292101s) [seg_pos={0 0}, upload=576429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:28:51.634294529 (657h59m51.505377564s) [upload=576429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:28:51.634313756 (657h59m51.505396791s) [upload=576429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:28:51.926636932 (657h59m51.797719967s) [upload=576429]: Done uploading segments... Jan 2 01:28:51.926660708 (657h59m51.797743753s) [upload=576429]: Upload finished. err: <nil> Duration: 22.164970687s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.577274785s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:34:29 UTC) orbiter warning: context deadline exceeded Duration: 3.577311774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 169.284126ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:34:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 169.314685ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 3.495025752s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:34:23 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.49505593s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 154.790629ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:33:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 154.818192ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X..X....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X.....] duration: 6.511111927s trace: https://jaeger.dev.storj.io/trace/5f2ae0ac088f21fb FAIL (2026-01-02 01:33:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q) context deadline exceeded ======LOGS====== Jan 2 01:28:48.865298164 (657h59m48.736381199s) [upload=576429]: Starting upload Jan 2 01:28:48.865409055 (657h59m48.73649209s) [upload=576429]: Done starting upload Jan 2 01:28:48.865527081 (657h59m48.736610126s) [upload=576429]: Uploading segments... Jan 2 01:28:48.865619115 (657h59m48.73670215s) [upload=576429]: Got next segment. Inline: false Jan 2 01:28:48.865659492 (657h59m48.736742537s) [upload=576429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:28:48.865666045 (657h59m48.73674908s) [upload=576429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:28:49.108468076 (657h59m48.979551122s) [seg_pos={0 0}, upload=576429]: Begin upload segment... Jan 2 01:28:49.108512270 (657h59m48.979595305s) [seg_pos={0 0}, upload=576429]: Done begin upload segment. Jan 2 01:28:49.108557808 (657h59m48.979640863s) [noise=true , address=145.79.227.16:4912 , piece_id=PDGWWSRYIFHLH426P7FWIJHYQOVYVFVK2EQPAB4NCMBNEY2A632A, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.108583075 (657h59m48.97966612s) [noise=true , address=145.79.227.18:4007 , piece_id=WBCAAEZDZR4ROB4TX5GMS5INGZ5ZNYUM6D6T4VCE5GV2CXZLUC5A, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.109078840 (657h59m48.980161895s) [noise=true , address=145.79.227.37:4566 , piece_id=GAK64W3EPYBKQSJC6CZO3SSTS3XXUPDWSWALIACBZ4GPLIWNSS5Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.109498178 (657h59m48.980581233s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.110068083 (657h59m48.981151128s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.110529702 (657h59m48.981612747s) [noise=true , address=145.79.227.27:4690 , piece_id=JWZ2PKIOHGGB223CPLR77QWSQW5S3VVRKJKVF5UNJPTLZIYNCZLA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.111011129 (657h59m48.982094174s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.112367001 (657h59m48.983450036s) [noise=true , address=145.79.227.7:4294 , piece_id=W3BDYLEBEUII2FFWAD4ODWQ4YLF4IJJT3UF5JLZ23VUHHEIFWKWA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.112743067 (657h59m48.983826112s) [noise=true , address=45.91.116.5:4299 , piece_id=KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.113497813 (657h59m48.984580859s) [noise=true , address=77.90.31.17:4770 , piece_id=WQYNR72547D7ORNETLVURASVFQS237H67EXANZGZE4BWW4ERRVEQ, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.114878823 (657h59m48.985961869s) [noise=true , address=145.79.227.15:4876 , piece_id=KJLPSVOG4T6ABBYRQAC5ODG6U5OKQC6THJK7JGL7FQHZ62MDACWA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.116961329 (657h59m48.988044384s) [noise=true , address=145.79.227.52:4447 , piece_id=DYNXASJFMQ72IQ3KTPMEH2L5FPXXCXWQTCMIPZN3ZANAHNRA6DIA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.118513995 (657h59m48.98959704s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.120245753 (657h59m48.991328798s) [noise=true , address=145.79.227.48:4303 , piece_id=WQ7QVUF2CC2E23SQFFYTFDT7VPVHZFCAIP4P7A6EMR7CIQGMTLOA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.120462736 (657h59m48.991545781s) [noise=true , address=145.79.227.5:4221 , piece_id=4RU3WR7YTQD6W5I47ATFSHKLCYK3CX5DJOAX4VB7NQ24NSVLBSJA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.121557330 (657h59m48.992640365s) [noise=true , address=77.90.31.13:4626 , piece_id=UWAE4R62SWC2X6H56QZE7RS4OBEUWOLEXDCXFTLIZ2JFNHTXVKOQ, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:50.815693191 (657h59m50.686776226s) [upload=576429]: Next returned nil segment Jan 2 01:28:50.815711606 (657h59m50.686794642s) [upload=576429]: Waiting for error group managing segments... Jan 2 01:28:51.105308571 (657h59m50.976391616s) [noise=true , address=145.79.227.16:4912 , piece_id=PDGWWSRYIFHLH426P7FWIJHYQOVYVFVK2EQPAB4NCMBNEY2A632A, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.106445306 (657h59m50.977528341s) [noise=true , address=145.79.227.37:4566 , piece_id=GAK64W3EPYBKQSJC6CZO3SSTS3XXUPDWSWALIACBZ4GPLIWNSS5Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.106496463 (657h59m50.977579508s) [noise=true , address=145.79.227.5:4221 , piece_id=4RU3WR7YTQD6W5I47ATFSHKLCYK3CX5DJOAX4VB7NQ24NSVLBSJA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.114999614 (657h59m50.986082659s) [noise=true , address=145.79.227.52:4447 , piece_id=DYNXASJFMQ72IQ3KTPMEH2L5FPXXCXWQTCMIPZN3ZANAHNRA6DIA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.115736637 (657h59m50.986819682s) [noise=true , address=145.79.227.48:4303 , piece_id=WQ7QVUF2CC2E23SQFFYTFDT7VPVHZFCAIP4P7A6EMR7CIQGMTLOA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.116229075 (657h59m50.98731212s) [noise=true , address=145.79.227.27:4690 , piece_id=JWZ2PKIOHGGB223CPLR77QWSQW5S3VVRKJKVF5UNJPTLZIYNCZLA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.129635131 (657h59m51.000718176s) [noise=true , address=145.79.227.15:4876 , piece_id=KJLPSVOG4T6ABBYRQAC5ODG6U5OKQC6THJK7JGL7FQHZ62MDACWA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.136267359 (657h59m51.007350395s) [noise=true , address=145.79.227.7:4294 , piece_id=W3BDYLEBEUII2FFWAD4ODWQ4YLF4IJJT3UF5JLZ23VUHHEIFWKWA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.250622625 (657h59m51.12170566s) [noise=true , address=145.79.227.18:4007 , piece_id=WBCAAEZDZR4ROB4TX5GMS5INGZ5ZNYUM6D6T4VCE5GV2CXZLUC5A, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.364460585 (657h59m51.23554363s) [noise=true , address=77.90.31.13:4626 , piece_id=UWAE4R62SWC2X6H56QZE7RS4OBEUWOLEXDCXFTLIZ2JFNHTXVKOQ, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.376934517 (657h59m51.248017552s) [noise=true , address=77.90.31.17:4770 , piece_id=WQYNR72547D7ORNETLVURASVFQS237H67EXANZGZE4BWW4ERRVEQ, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.613301886 (657h59m51.484384931s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.633527909 (657h59m51.504610954s) [noise=true , address=45.91.116.5:4299 , piece_id=KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.633550943 (657h59m51.504633988s) [seg_pos={0 0}, upload=576429]: Segment reached optimal threshold of 13 pieces. Jan 2 01:28:51.633771082 (657h59m51.504854127s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection Jan 2 01:28:51.633781091 (657h59m51.504864126s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.633851505 (657h59m51.50493454s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Jan 2 01:28:51.633861725 (657h59m51.50494477s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.634178087 (657h59m51.505261132s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Jan 2 01:28:51.634187896 (657h59m51.505270931s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.634209056 (657h59m51.505292101s) [seg_pos={0 0}, upload=576429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:28:51.634294529 (657h59m51.505377564s) [upload=576429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:28:51.634313756 (657h59m51.505396791s) [upload=576429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:28:51.926636932 (657h59m51.797719967s) [upload=576429]: Done uploading segments... Jan 2 01:28:51.926660708 (657h59m51.797743753s) [upload=576429]: Upload finished. err: <nil> Duration: 6.511160558s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 3.281102139s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:33:14 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.281126455s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.563342236s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:33:14 UTC) orbiter warning: context deadline exceeded Duration: 3.563368436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.X..X....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X....] duration: 5.915259118s trace: https://jaeger.dev.storj.io/trace/5f2ae0ac088f21fb FAIL (2026-01-02 01:33:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q) context deadline exceeded ======LOGS====== Jan 2 01:28:48.865298164 (657h59m48.736381199s) [upload=576429]: Starting upload Jan 2 01:28:48.865409055 (657h59m48.73649209s) [upload=576429]: Done starting upload Jan 2 01:28:48.865527081 (657h59m48.736610126s) [upload=576429]: Uploading segments... Jan 2 01:28:48.865619115 (657h59m48.73670215s) [upload=576429]: Got next segment. Inline: false Jan 2 01:28:48.865659492 (657h59m48.736742537s) [upload=576429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:28:48.865666045 (657h59m48.73674908s) [upload=576429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:28:49.108468076 (657h59m48.979551122s) [seg_pos={0 0}, upload=576429]: Begin upload segment... Jan 2 01:28:49.108512270 (657h59m48.979595305s) [seg_pos={0 0}, upload=576429]: Done begin upload segment. Jan 2 01:28:49.108557808 (657h59m48.979640863s) [noise=true , address=145.79.227.16:4912 , piece_id=PDGWWSRYIFHLH426P7FWIJHYQOVYVFVK2EQPAB4NCMBNEY2A632A, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.108583075 (657h59m48.97966612s) [noise=true , address=145.79.227.18:4007 , piece_id=WBCAAEZDZR4ROB4TX5GMS5INGZ5ZNYUM6D6T4VCE5GV2CXZLUC5A, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.109078840 (657h59m48.980161895s) [noise=true , address=145.79.227.37:4566 , piece_id=GAK64W3EPYBKQSJC6CZO3SSTS3XXUPDWSWALIACBZ4GPLIWNSS5Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.109498178 (657h59m48.980581233s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.110068083 (657h59m48.981151128s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.110529702 (657h59m48.981612747s) [noise=true , address=145.79.227.27:4690 , piece_id=JWZ2PKIOHGGB223CPLR77QWSQW5S3VVRKJKVF5UNJPTLZIYNCZLA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.111011129 (657h59m48.982094174s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.112367001 (657h59m48.983450036s) [noise=true , address=145.79.227.7:4294 , piece_id=W3BDYLEBEUII2FFWAD4ODWQ4YLF4IJJT3UF5JLZ23VUHHEIFWKWA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.112743067 (657h59m48.983826112s) [noise=true , address=45.91.116.5:4299 , piece_id=KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.113497813 (657h59m48.984580859s) [noise=true , address=77.90.31.17:4770 , piece_id=WQYNR72547D7ORNETLVURASVFQS237H67EXANZGZE4BWW4ERRVEQ, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.114878823 (657h59m48.985961869s) [noise=true , address=145.79.227.15:4876 , piece_id=KJLPSVOG4T6ABBYRQAC5ODG6U5OKQC6THJK7JGL7FQHZ62MDACWA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.116961329 (657h59m48.988044384s) [noise=true , address=145.79.227.52:4447 , piece_id=DYNXASJFMQ72IQ3KTPMEH2L5FPXXCXWQTCMIPZN3ZANAHNRA6DIA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.118513995 (657h59m48.98959704s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.120245753 (657h59m48.991328798s) [noise=true , address=145.79.227.48:4303 , piece_id=WQ7QVUF2CC2E23SQFFYTFDT7VPVHZFCAIP4P7A6EMR7CIQGMTLOA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.120462736 (657h59m48.991545781s) [noise=true , address=145.79.227.5:4221 , piece_id=4RU3WR7YTQD6W5I47ATFSHKLCYK3CX5DJOAX4VB7NQ24NSVLBSJA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.121557330 (657h59m48.992640365s) [noise=true , address=77.90.31.13:4626 , piece_id=UWAE4R62SWC2X6H56QZE7RS4OBEUWOLEXDCXFTLIZ2JFNHTXVKOQ, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:50.815693191 (657h59m50.686776226s) [upload=576429]: Next returned nil segment Jan 2 01:28:50.815711606 (657h59m50.686794642s) [upload=576429]: Waiting for error group managing segments... Jan 2 01:28:51.105308571 (657h59m50.976391616s) [noise=true , address=145.79.227.16:4912 , piece_id=PDGWWSRYIFHLH426P7FWIJHYQOVYVFVK2EQPAB4NCMBNEY2A632A, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.106445306 (657h59m50.977528341s) [noise=true , address=145.79.227.37:4566 , piece_id=GAK64W3EPYBKQSJC6CZO3SSTS3XXUPDWSWALIACBZ4GPLIWNSS5Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.106496463 (657h59m50.977579508s) [noise=true , address=145.79.227.5:4221 , piece_id=4RU3WR7YTQD6W5I47ATFSHKLCYK3CX5DJOAX4VB7NQ24NSVLBSJA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.114999614 (657h59m50.986082659s) [noise=true , address=145.79.227.52:4447 , piece_id=DYNXASJFMQ72IQ3KTPMEH2L5FPXXCXWQTCMIPZN3ZANAHNRA6DIA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.115736637 (657h59m50.986819682s) [noise=true , address=145.79.227.48:4303 , piece_id=WQ7QVUF2CC2E23SQFFYTFDT7VPVHZFCAIP4P7A6EMR7CIQGMTLOA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.116229075 (657h59m50.98731212s) [noise=true , address=145.79.227.27:4690 , piece_id=JWZ2PKIOHGGB223CPLR77QWSQW5S3VVRKJKVF5UNJPTLZIYNCZLA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.129635131 (657h59m51.000718176s) [noise=true , address=145.79.227.15:4876 , piece_id=KJLPSVOG4T6ABBYRQAC5ODG6U5OKQC6THJK7JGL7FQHZ62MDACWA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.136267359 (657h59m51.007350395s) [noise=true , address=145.79.227.7:4294 , piece_id=W3BDYLEBEUII2FFWAD4ODWQ4YLF4IJJT3UF5JLZ23VUHHEIFWKWA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.250622625 (657h59m51.12170566s) [noise=true , address=145.79.227.18:4007 , piece_id=WBCAAEZDZR4ROB4TX5GMS5INGZ5ZNYUM6D6T4VCE5GV2CXZLUC5A, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.364460585 (657h59m51.23554363s) [noise=true , address=77.90.31.13:4626 , piece_id=UWAE4R62SWC2X6H56QZE7RS4OBEUWOLEXDCXFTLIZ2JFNHTXVKOQ, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.376934517 (657h59m51.248017552s) [noise=true , address=77.90.31.17:4770 , piece_id=WQYNR72547D7ORNETLVURASVFQS237H67EXANZGZE4BWW4ERRVEQ, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.613301886 (657h59m51.484384931s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.633527909 (657h59m51.504610954s) [noise=true , address=45.91.116.5:4299 , piece_id=KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.633550943 (657h59m51.504633988s) [seg_pos={0 0}, upload=576429]: Segment reached optimal threshold of 13 pieces. Jan 2 01:28:51.633771082 (657h59m51.504854127s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection Jan 2 01:28:51.633781091 (657h59m51.504864126s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.633851505 (657h59m51.50493454s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Jan 2 01:28:51.633861725 (657h59m51.50494477s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.634178087 (657h59m51.505261132s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Jan 2 01:28:51.634187896 (657h59m51.505270931s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.634209056 (657h59m51.505292101s) [seg_pos={0 0}, upload=576429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:28:51.634294529 (657h59m51.505377564s) [upload=576429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:28:51.634313756 (657h59m51.505396791s) [upload=576429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:28:51.926636932 (657h59m51.797719967s) [upload=576429]: Done uploading segments... Jan 2 01:28:51.926660708 (657h59m51.797743753s) [upload=576429]: Upload finished. err: <nil> Duration: 5.915295466s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 200.396136ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:33:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 200.429271ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.611669098s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:32:42 UTC) orbiter warning: context deadline exceeded Duration: 3.61168589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 164.666362ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:32:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 164.712719ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.433750648s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:32:24 UTC) orbiter warning: context deadline exceeded Duration: 3.433799612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.23198142s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:32:01 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.232016057s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX.X..X....XXX....X.XX...X..X..XXXX....X....XXX....XXX.X...] duration: 6.211769946s trace: https://jaeger.dev.storj.io/trace/5f2ae0ac088f21fb FAIL (2026-01-02 01:31:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q) context deadline exceeded ======LOGS====== Jan 2 01:28:48.865298164 (657h59m48.736381199s) [upload=576429]: Starting upload Jan 2 01:28:48.865409055 (657h59m48.73649209s) [upload=576429]: Done starting upload Jan 2 01:28:48.865527081 (657h59m48.736610126s) [upload=576429]: Uploading segments... Jan 2 01:28:48.865619115 (657h59m48.73670215s) [upload=576429]: Got next segment. Inline: false Jan 2 01:28:48.865659492 (657h59m48.736742537s) [upload=576429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:28:48.865666045 (657h59m48.73674908s) [upload=576429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:28:49.108468076 (657h59m48.979551122s) [seg_pos={0 0}, upload=576429]: Begin upload segment... Jan 2 01:28:49.108512270 (657h59m48.979595305s) [seg_pos={0 0}, upload=576429]: Done begin upload segment. Jan 2 01:28:49.108557808 (657h59m48.979640863s) [noise=true , address=145.79.227.16:4912 , piece_id=PDGWWSRYIFHLH426P7FWIJHYQOVYVFVK2EQPAB4NCMBNEY2A632A, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.108583075 (657h59m48.97966612s) [noise=true , address=145.79.227.18:4007 , piece_id=WBCAAEZDZR4ROB4TX5GMS5INGZ5ZNYUM6D6T4VCE5GV2CXZLUC5A, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.109078840 (657h59m48.980161895s) [noise=true , address=145.79.227.37:4566 , piece_id=GAK64W3EPYBKQSJC6CZO3SSTS3XXUPDWSWALIACBZ4GPLIWNSS5Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.109498178 (657h59m48.980581233s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.110068083 (657h59m48.981151128s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.110529702 (657h59m48.981612747s) [noise=true , address=145.79.227.27:4690 , piece_id=JWZ2PKIOHGGB223CPLR77QWSQW5S3VVRKJKVF5UNJPTLZIYNCZLA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.111011129 (657h59m48.982094174s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.112367001 (657h59m48.983450036s) [noise=true , address=145.79.227.7:4294 , piece_id=W3BDYLEBEUII2FFWAD4ODWQ4YLF4IJJT3UF5JLZ23VUHHEIFWKWA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.112743067 (657h59m48.983826112s) [noise=true , address=45.91.116.5:4299 , piece_id=KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.113497813 (657h59m48.984580859s) [noise=true , address=77.90.31.17:4770 , piece_id=WQYNR72547D7ORNETLVURASVFQS237H67EXANZGZE4BWW4ERRVEQ, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.114878823 (657h59m48.985961869s) [noise=true , address=145.79.227.15:4876 , piece_id=KJLPSVOG4T6ABBYRQAC5ODG6U5OKQC6THJK7JGL7FQHZ62MDACWA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.116961329 (657h59m48.988044384s) [noise=true , address=145.79.227.52:4447 , piece_id=DYNXASJFMQ72IQ3KTPMEH2L5FPXXCXWQTCMIPZN3ZANAHNRA6DIA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.118513995 (657h59m48.98959704s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.120245753 (657h59m48.991328798s) [noise=true , address=145.79.227.48:4303 , piece_id=WQ7QVUF2CC2E23SQFFYTFDT7VPVHZFCAIP4P7A6EMR7CIQGMTLOA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.120462736 (657h59m48.991545781s) [noise=true , address=145.79.227.5:4221 , piece_id=4RU3WR7YTQD6W5I47ATFSHKLCYK3CX5DJOAX4VB7NQ24NSVLBSJA, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:49.121557330 (657h59m48.992640365s) [noise=true , address=77.90.31.13:4626 , piece_id=UWAE4R62SWC2X6H56QZE7RS4OBEUWOLEXDCXFTLIZ2JFNHTXVKOQ, seg_pos={0 0}, upload=576429]: Uploading piece... Jan 2 01:28:50.815693191 (657h59m50.686776226s) [upload=576429]: Next returned nil segment Jan 2 01:28:50.815711606 (657h59m50.686794642s) [upload=576429]: Waiting for error group managing segments... Jan 2 01:28:51.105308571 (657h59m50.976391616s) [noise=true , address=145.79.227.16:4912 , piece_id=PDGWWSRYIFHLH426P7FWIJHYQOVYVFVK2EQPAB4NCMBNEY2A632A, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.106445306 (657h59m50.977528341s) [noise=true , address=145.79.227.37:4566 , piece_id=GAK64W3EPYBKQSJC6CZO3SSTS3XXUPDWSWALIACBZ4GPLIWNSS5Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.106496463 (657h59m50.977579508s) [noise=true , address=145.79.227.5:4221 , piece_id=4RU3WR7YTQD6W5I47ATFSHKLCYK3CX5DJOAX4VB7NQ24NSVLBSJA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.114999614 (657h59m50.986082659s) [noise=true , address=145.79.227.52:4447 , piece_id=DYNXASJFMQ72IQ3KTPMEH2L5FPXXCXWQTCMIPZN3ZANAHNRA6DIA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.115736637 (657h59m50.986819682s) [noise=true , address=145.79.227.48:4303 , piece_id=WQ7QVUF2CC2E23SQFFYTFDT7VPVHZFCAIP4P7A6EMR7CIQGMTLOA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.116229075 (657h59m50.98731212s) [noise=true , address=145.79.227.27:4690 , piece_id=JWZ2PKIOHGGB223CPLR77QWSQW5S3VVRKJKVF5UNJPTLZIYNCZLA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.129635131 (657h59m51.000718176s) [noise=true , address=145.79.227.15:4876 , piece_id=KJLPSVOG4T6ABBYRQAC5ODG6U5OKQC6THJK7JGL7FQHZ62MDACWA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.136267359 (657h59m51.007350395s) [noise=true , address=145.79.227.7:4294 , piece_id=W3BDYLEBEUII2FFWAD4ODWQ4YLF4IJJT3UF5JLZ23VUHHEIFWKWA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.250622625 (657h59m51.12170566s) [noise=true , address=145.79.227.18:4007 , piece_id=WBCAAEZDZR4ROB4TX5GMS5INGZ5ZNYUM6D6T4VCE5GV2CXZLUC5A, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.364460585 (657h59m51.23554363s) [noise=true , address=77.90.31.13:4626 , piece_id=UWAE4R62SWC2X6H56QZE7RS4OBEUWOLEXDCXFTLIZ2JFNHTXVKOQ, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.376934517 (657h59m51.248017552s) [noise=true , address=77.90.31.17:4770 , piece_id=WQYNR72547D7ORNETLVURASVFQS237H67EXANZGZE4BWW4ERRVEQ, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.613301886 (657h59m51.484384931s) [noise=true , address=45.91.116.8:4046 , piece_id=Q6GIJPTAL22FYEMOAEZVZNY66YMSWJLLXFWQDZ2VDDCE5YAGLQ4Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.633527909 (657h59m51.504610954s) [noise=true , address=45.91.116.5:4299 , piece_id=KSQGMIAHJ3TJQK6JTBMN37U43YDMWVGQWCNH2PTEHV2VXJ4VH5LA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: <nil> Jan 2 01:28:51.633550943 (657h59m51.504633988s) [seg_pos={0 0}, upload=576429]: Segment reached optimal threshold of 13 pieces. Jan 2 01:28:51.633771082 (657h59m51.504854127s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57900->77.90.31.14:4663: use of closed network connection Jan 2 01:28:51.633781091 (657h59m51.504864126s) [noise=true , address=77.90.31.14:4663 , piece_id=IGURFLFO4WH6I4TE66PHJYHTT32WUECMJOR64ZIM67AMAABILWZA, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.633851505 (657h59m51.50493454s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Jan 2 01:28:51.633861725 (657h59m51.50494477s) [noise=true , address=145.79.227.44:4194 , piece_id=RWYYR4GVA5Z2EHGP47LND6BF4UB2VKOAS4MDXYIOPXYABOUG3R4Q, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.634178087 (657h59m51.505261132s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Jan 2 01:28:51.634187896 (657h59m51.505270931s) [noise=true , address=77.90.31.10:4517 , piece_id=KTNKLBRYMRJSMMAOSPBB4747F2PXQPUATPTYCJ47XRGVB5XYJFAA, seg_pos={0 0}, upload=576429]: Long tail cancellation due to optimal threshold Jan 2 01:28:51.634209056 (657h59m51.505292101s) [seg_pos={0 0}, upload=576429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:28:51.634294529 (657h59m51.505377564s) [upload=576429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:28:51.634313756 (657h59m51.505396791s) [upload=576429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:28:51.926636932 (657h59m51.797719967s) [upload=576429]: Done uploading segments... Jan 2 01:28:51.926660708 (657h59m51.797743753s) [upload=576429]: Upload finished. err: <nil> Duration: 6.211806046s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.592259707s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:31:46 UTC) orbiter warning: context deadline exceeded Duration: 3.592288772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 170.635749ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:31:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 170.668291ms core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)