Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 860326 reports (Page 11768 of 17207)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.73215169s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:51:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.732194553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........X...X....X..............................XX......X.X.] duration: 5.725021302s trace: https://jaeger.dev.storj.io/trace/7ec66c9dfb3ce3e5 FAIL (2025-12-07 23:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.725038796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........X...X....X..............................XX......X.X] duration: 1m0.001884418s trace: https://jaeger.dev.storj.io/trace/7ec66c9dfb3ce3e5 FAIL (2025-12-07 23:51:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001906692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X.............X................X.........] duration: 3.444133813s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:50:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444158149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.778707661s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:50:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.778734963s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X.............X................X........] duration: 3.546230333s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:50:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54625444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.675980541s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:49:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.676007964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........X...X....X..............................XX......X.] duration: 5.904563077s trace: https://jaeger.dev.storj.io/trace/5d9ed230c6ccf7b7 FAIL (2025-12-07 23:49:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.904579618s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........X...X....X..............................XX......X] duration: 1m0.000982934s trace: https://jaeger.dev.storj.io/trace/5d9ed230c6ccf7b7 FAIL (2025-12-07 23:49:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001008703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X.............X................X.......] duration: 3.476888463s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:49:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476916847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.789444082s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:48:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.789474119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.763637233s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:48:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.763666769s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.522501246s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:47:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.52254547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X.............X................X......] duration: 3.474388856s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:47:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474414555s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............X...X....X..............................XX......] duration: 5.928568757s trace: https://jaeger.dev.storj.io/trace/2666551e0c9ff9dd FAIL (2025-12-07 23:47:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.92858632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.69284929s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:47:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.692870971s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.777498043s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:46:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.777562155s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X.............X................X.....] duration: 3.484818804s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:46:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48484178s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.791255581s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:46:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.791284796s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............X...X....X..............................XX.....] duration: 5.66745846s trace: https://jaeger.dev.storj.io/trace/2666551e0c9ff9dd FAIL (2025-12-07 23:46:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.667475954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.69759577s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:45:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.697623424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X.............X................X....] duration: 3.371982999s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:45:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.372045089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.566350847s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:45:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.566376977s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.333752594s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:45:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.333784924s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.780057818s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:45:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.780089188s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X...X....X..............................XX....] duration: 5.895591623s trace: https://jaeger.dev.storj.io/trace/2666551e0c9ff9dd FAIL (2025-12-07 23:45:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.895619095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.691767151s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:45:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.691804571s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X.............X................X...] duration: 3.487881048s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:45:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487911255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.604565294s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:44:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.604596072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.719704637s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:44:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.71973301s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............X...X....X..............................XX...] duration: 5.839536584s trace: https://jaeger.dev.storj.io/trace/2666551e0c9ff9dd FAIL (2025-12-07 23:44:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.839596698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.658670479s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:44:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.65869734s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X.............X................X..] duration: 3.482036959s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:44:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.482119866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X...............X...X....X..............................XX..] duration: 5.946709907s trace: https://jaeger.dev.storj.io/trace/2666551e0c9ff9dd FAIL (2025-12-07 23:44:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.946754633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.69542417s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:44:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.695456282s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.654138839s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:43:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.654165622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.719852096s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:43:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.719878756s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.756692722s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:43:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.756761553s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X...............X...X....X..............................XX.] duration: 5.656766544s trace: https://jaeger.dev.storj.io/trace/2666551e0c9ff9dd FAIL (2025-12-07 23:43:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.656780871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X...............X...X....X..............................XX] duration: 1m0.001657322s trace: https://jaeger.dev.storj.io/trace/2666551e0c9ff9dd FAIL (2025-12-07 23:43:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001724721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X.............X................X.] duration: 3.5316342s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:43:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531661442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.221140332s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:43:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.221177061s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.662623223s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:43:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.662649635s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................X.............X................X] duration: 5.00086699s trace: https://jaeger.dev.storj.io/trace/6d7da843d28aaca FAIL (2025-12-07 23:42:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000888851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.854093846s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:42:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.854122018s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X...............X...X....X..............................X] duration: 1m0.001481619s trace: https://jaeger.dev.storj.io/trace/4a423f4ac85de6c5 FAIL (2025-12-07 23:42:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001501007s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X.............X................] duration: 3.503487634s trace: https://jaeger.dev.storj.io/trace/e4a8812130e0d1c FAIL (2025-12-07 23:42:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503509647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.71738484s trace: https://jaeger.dev.storj.io/trace/705e533ea54dbd2e FAIL (2025-12-07 23:42:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.71751133s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.7719403s trace: https://jaeger.dev.storj.io/trace/7af41f8f7f3456f4 FAIL (2025-12-07 23:42:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.556741966 (55h29m5.427825011s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.557760355 (55h29m5.4288434s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.558611125 (55h29m5.42969419s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559231146 (55h29m5.430314191s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.559842449 (55h29m5.430925484s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.560479904 (55h29m5.431562939s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.561118028 (55h29m5.432201073s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.562981156 (55h29m5.434064201s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.564678598 (55h29m5.435761643s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:05.579345735 (55h29m5.45042877s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Uploading piece... Dec 7 22:58:06.380019068 (55h29m6.251102113s) [upload=48765]: Next returned nil segment Dec 7 22:58:06.380037754 (55h29m6.251120799s) [upload=48765]: Waiting for error group managing segments... Dec 7 22:58:06.773287581 (55h29m6.644370616s) [noise=true , address=185.167.204.206:4063 , piece_id=6MXKN2JJPK52KNF36QLNF3XDCDTVSGHQAJDKYO2WEL7BWRTABC5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774227821 (55h29m6.645310866s) [noise=true , address=185.167.204.205:4021 , piece_id=JXUWY2LWGJGU655Y7SQMBRCSQYTUBRNRSALZ4FKLSTEJQ7KT73AA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.774818546 (55h29m6.645901581s) [noise=true , address=185.167.204.206:4069 , piece_id=GSJ2BS7MTUE6RDMWA5PBYGZ47KV53RDK5CJFV7HPYTR4LIKX57DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.775818009 (55h29m6.646901084s) [noise=true , address=185.167.204.205:4030 , piece_id=KYZ6C6LHVDBUWG7ATJ5GK7WPQTGCO47RPVDCUK6P7YSDAEGAKJVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.776946959 (55h29m6.648030004s) [noise=true , address=185.167.204.205:4013 , piece_id=PQREC457Q7KXCUESC2DUFJ32CRM3UXMOPR4FNYYXRFEOYG275XOQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.778951605 (55h29m6.650034641s) [noise=true , address=185.167.204.206:4043 , piece_id=JWXOBMPCBK3WJ22KJBLFV7SUVEFWKY6LX7VX6CVDZUKUFFYGCN6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.779607304 (55h29m6.650690349s) [noise=true , address=185.167.204.206:4064 , piece_id=5G5T7WTOOCBRQQJVK3GVBOKC25UV4LSHG7VQYOWSWBFMLQEHEXVQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.785729331 (55h29m6.656812366s) [noise=true , address=185.167.204.206:4045 , piece_id=ZOS6YE4NP7MFGNAI7FRAPCDH7TIKEG5YAS7OGUPLDAPQIGT6F3JQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.787367650 (55h29m6.658450695s) [noise=true , address=185.167.204.205:4018 , piece_id=GLEKC57OFTYOLCPGP25IVJBY5EGTFAUJGR7RFAFV3HGU4UFB6UQQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788378124 (55h29m6.659461169s) [noise=true , address=185.167.204.206:4046 , piece_id=NOV6T6ZE2QEETGXSHFK4OC2ZYMUDMMA7CNH7D47Q3GJLVEZJLR6Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788431557 (55h29m6.659514592s) [noise=true , address=185.167.204.206:4057 , piece_id=WSVDIZHML7JOAMXLY5HA6EC6LK4EKDWDSASCFCFU6AUHM4USZVCQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.788864150 (55h29m6.659947195s) [noise=true , address=185.167.204.206:4054 , piece_id=GYIO75YOAW5VRIIMB4G6GFSVLQRWH5RO66IU6TYKMU345XJODZIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:06.801527192 (55h29m6.672610237s) [noise=true , address=185.167.204.205:4032 , piece_id=7YJWM4YYYICLYTPGFAPPF5AMSQO6TGKRG74XRYE764QWUGXVKH5Q, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.338558042 (55h29m7.209641077s) [noise=true , address=81.171.14.70:4036 , piece_id=GHV6GUZBJCTBTGXABZNGOLMLT6KWD7BVJMJZXNG6NDJIAYKVEYGQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.391621471 (55h29m7.262704506s) [noise=true , address=81.171.14.70:4029 , piece_id=ZHSRZNTSCQ55VCAWUSAFN6E5FI3XVCVRCPPGFL3FNHHI3KXUOCLQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.396060443 (55h29m7.267143478s) [noise=true , address=81.171.14.70:4027 , piece_id=KUFNR3R4S4R3GEOFNNFHWL5ZXAKGVAKKKZK372RI57GYMC5X7LKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.401160444 (55h29m7.272243479s) [noise=true , address=81.171.14.70:4039 , piece_id=OCIFXFUH2TKJYBT3DVBM5THZQT47HE53DJSJNF4NNZNZBRII5GMA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.438501735 (55h29m7.30958477s) [noise=true , address=81.171.14.70:4041 , piece_id=L22C75QVTZTLL6IE66EMMX26M2JZBYAL2IOTMEDQTQQUOB7BALKA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.454249140 (55h29m7.325332175s) [noise=true , address=81.171.14.70:4035 , piece_id=IF6HZRPWIGWSN2BRXI3NTGWXRPP2VKBNMADTHMBGAVQ5I3H5JSUQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.508014374 (55h29m7.379097419s) [noise=true , address=81.171.14.70:4044 , piece_id=UXPAOYQRWDXAD2PZ4WGGBMGOYYNHVVV7RD5R6RA5V26U3TE2YDQA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.828013886 (55h29m7.699096931s) [noise=true , address=81.171.14.70:4031 , piece_id=L2X3FA5MU5WXS2CLHNSLOOJPO2MASF5G4SWUKGRUSTUCKYK7FXDQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.839373846 (55h29m7.710456891s) [noise=true , address=81.171.14.70:4043 , piece_id=IVSFI6IUTYG2QXP7J6BW6JBQASYXCHKAB57Q6GEVMWYOBGTII75A, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.968046842 (55h29m7.839129887s) [noise=true , address=89.149.201.147:4011 , piece_id=N4CKZNH4IMEUCMOH4JH6X7VGUGIEQNM2VVXIARICS3JPKLMY4HUA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:07.985511697 (55h29m7.856594742s) [noise=true , address=89.149.201.147:4017 , piece_id=N7KUIZHFT3X6M73UINWKYVVDHKCHQE2YHTCLPZZKNCDW5ESRNUGA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.000651885 (55h29m7.87173494s) [noise=true , address=89.149.201.147:4013 , piece_id=IWGKXTU6WCJWYJWPLVXEX5PQVXH7CQHBL2HXALPXPLPXOWIU37DA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022779458 (55h29m7.893862504s) [noise=true , address=89.149.201.147:4020 , piece_id=DVGDAZ4MLDSYENLSO4XCZO6TS5O2TRAEYVZBU52MJKBNQRLVC5BA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: <nil> Dec 7 22:58:08.022799838 (55h29m7.893882873s) [seg_pos={0 0}, upload=48765]: Segment reached optimal threshold of 26 pieces. Dec 7 22:58:08.022989659 (55h29m7.894072704s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.022996782 (55h29m7.894079827s) [noise=true , address=89.149.201.147:4012 , piece_id=LWGTR5TTX2MLXULGMZIPO5JKEPAKYPTYEAGBB63Z5RTTVR2TOYIA, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023322272 (55h29m7.894405338s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 7 22:58:08.023345656 (55h29m7.894428711s) [noise=true , address=89.149.201.147:4019 , piece_id=NYRQD3DHDXA4HVL5GRYFG6JIB4SYFOVTBZDIHYQL2DT2FUPO7JEQ, seg_pos={0 0}, upload=48765]: Long tail cancellation due to optimal threshold Dec 7 22:58:08.023386635 (55h29m7.89446967s) [seg_pos={0 0}, upload=48765]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 7 22:58:08.023537632 (55h29m7.894620667s) [upload=48765]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 22:58:08.023544135 (55h29m7.89462718s) [upload=48765]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:03:49.637653371 (55h34m49.508736406s) [upload=48765]: Done uploading segments... Dec 7 23:03:49.637698487 (55h34m49.508781522s) [upload=48765]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.771967662s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.762180126s trace: https://jaeger.dev.storj.io/trace/66ad62c5c711446b FAIL (2025-12-07 23:42:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207240803 (55h54m18.078323848s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.207996272 (55h54m18.079079317s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208445797 (55h54m18.079528842s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.208985926 (55h54m18.080068961s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209419341 (55h54m18.080502386s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.209906819 (55h54m18.080989864s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210409617 (55h54m18.081492662s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.210891805 (55h54m18.08197484s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.211370787 (55h54m18.082453832s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.213302946 (55h54m18.084385981s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.216593051 (55h54m18.087676096s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.218485725 (55h54m18.08956876s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.221734531 (55h54m18.092817576s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.226519332 (55h54m18.097602367s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.231814655 (55h54m18.1028977s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.234302423 (55h54m18.105385468s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:18.237490443 (55h54m18.108573488s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Uploading piece... Dec 7 23:23:19.531237398 (55h54m19.402320433s) [upload=49144]: Next returned nil segment Dec 7 23:23:19.531251114 (55h54m19.402334159s) [upload=49144]: Waiting for error group managing segments... Dec 7 23:23:19.866004356 (55h54m19.737087391s) [noise=true , address=193.149.164.143:4696 , piece_id=YYKTVQOHVNVC6U2JLKLVMHB7ZSNVUCX5KMV7GWLAM7XNQ7QZ3UEQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:19.963385365 (55h54m19.83446841s) [noise=true , address=64.49.13.154:4932 , piece_id=K3GBHL7BVBZKD24LU7YG6TYKXCRNNZRM4VTAJHNOTNRYJL5D74GQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.062063714 (55h54m19.933146749s) [noise=true , address=145.79.227.35:4493 , piece_id=WC4N7RZRLZDACYBIZLRPPAIMB3WMY5LPU5GIQZ66NZ2JX2MNEIOQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.073916684 (55h54m19.944999729s) [noise=true , address=145.79.227.47:4290 , piece_id=ZUKB7GPDRX6XNKRL3K2MJWOXBCN2KZNYZNYVJJIAGKBTL5PT3ZBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.085989733 (55h54m19.957072768s) [noise=true , address=145.79.227.39:4025 , piece_id=5O4MK4RC6YCNYD2P6GXBPRYBBZS2MU3G6BI4RYE6UCCKRYHE4NGA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.094959302 (55h54m19.966042337s) [noise=true , address=145.79.227.19:4084 , piece_id=AFNKCPFOUDPMDA2ZFUZLQLKPOAMWLK3RFBSNCCBECCJKYEKD5LUA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.124025920 (55h54m19.995108955s) [noise=true , address=145.79.227.5:4242 , piece_id=TLFGWMM3YGRHDGFZYMK7SYNKNP75YYZMCWLGRB2HGKYGBNURRJBQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.171858851 (55h54m20.042941886s) [noise=true , address=77.90.31.15:4703 , piece_id=CXU42ALT46NZJAJ66LYQ5VXVKIYAXXZADICFIHP56GTP7SBZ3DLA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.185288393 (55h54m20.056371428s) [noise=true , address=145.79.227.13:4823 , piece_id=VZPZJ353VYNCTRSRRXDXOM2Y7Y6DZMM32O7K46CECVSXLECTUUHA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.195072885 (55h54m20.06615592s) [noise=true , address=77.90.31.21:4897 , piece_id=5XWOADDY7PY2EHCXYGVWD47BOONX7R34WU4OPSXRBDUFWVAQ6NGQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.198943895 (55h54m20.070026941s) [noise=true , address=77.90.31.11:4587 , piece_id=OBXXOW4SJVEQU2NQCIDBE2GACU4XQ64R7UBP6UU2WYNQAIGPPR3Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.202198703 (55h54m20.073281738s) [noise=true , address=77.90.31.13:4657 , piece_id=4KAJPB3KOLGBGJTL3J7ZNRWV5YBEA4MJUCKIWGN7GBFANFHOPVRA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.220297716 (55h54m20.091380751s) [noise=true , address=77.90.31.10:4545 , piece_id=4K22IR566MB3EI3AJ4IJHFNOREKJRTMSHASDG345PWDSLLL4IPEA, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247379384 (55h54m20.118462419s) [noise=true , address=77.90.31.17:4783 , piece_id=GDGG366QEAHRJCPKK4C4BC374UPZGCP7B2ZFCFSS245ZBH62IKVQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: <nil> Dec 7 23:23:20.247397499 (55h54m20.118480534s) [seg_pos={0 0}, upload=49144]: Segment reached optimal threshold of 14 pieces. Dec 7 23:23:20.247674847 (55h54m20.118757882s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bzkWUwPDtuJDFyubHfhxxTxaw2M1tWxdeMbBF6AE5csFSmtsc): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247682701 (55h54m20.118765736s) [noise=true , address=45.91.116.7:4023 , piece_id=6HJHFJHWVUT4TFP4EFHGTQ77YCHQKU6RO7YQTFA66YTG5IP34S5Q, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247734029 (55h54m20.118817064s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247757424 (55h54m20.118840469s) [noise=true , address=45.91.116.14:4207 , piece_id=IKCH7KBNCVBPGYL6K6OFUEW3HURZTT44AKVVJZRKSSJLK73VIFMQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247805285 (55h54m20.11888832s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247813381 (55h54m20.118896416s) [noise=true , address=45.91.116.6:4981 , piece_id=UP64BFPJY7SE6OBMIWDDHKVTXPPKHPCTQZASLRDZ7DUEGDQUELSQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247907158 (55h54m20.118990193s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 7 23:23:20.247911868 (55h54m20.118994913s) [noise=true , address=145.79.227.33:4903 , piece_id=G6KCVO2SFL2XQOQFXQPM4QDQYBT6SV5CJLD3HPGK7O446M35QQJQ, seg_pos={0 0}, upload=49144]: Long tail cancellation due to optimal threshold Dec 7 23:23:20.247976701 (55h54m20.119059736s) [seg_pos={0 0}, upload=49144]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 7 23:23:20.248045772 (55h54m20.119128817s) [upload=49144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 23:23:20.248050342 (55h54m20.119133377s) [upload=49144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 23:25:57.835787371 (55h56m57.706870416s) [upload=49144]: Done uploading segments... Dec 7 23:25:57.835833379 (55h56m57.706916414s) [upload=49144]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.762211427s core uplink us-select-1 us1-placement-20

1
...
11766
11767
11768
11769
11770
...
17207