Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.XX.XXXXX.] duration: 4.854333079s trace: https://jaeger.dev.storj.io/trace/3c337605b3458717 FAIL (2025-12-17 14:35:29 UTC) orbiter warning: context deadline exceeded Duration: 4.854357976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.828735249s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:35:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.828795504s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX..X...X.X.XXXXXXXXXXX.X...XX.X...X....XXX.X.X.X.XXX.X..X.X] duration: 30.001802458s trace: https://jaeger.dev.storj.io/trace/3365dfcb62bf1979 FAIL (2025-12-17 14:35:17 UTC) orbiter warning: context deadline exceeded Duration: 30.001827077s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 12.128070479s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:35:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.12826002s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X...............X........X.....X.XX......................] duration: 9.790939767s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:35:07 UTC) orbiter warning: context deadline exceeded Duration: 9.790992488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.767830646s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:35:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.767862936s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 13.428078976s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:34:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 13.428102893s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X...X...............X........X.....X.XX.....................] duration: 23.299823846s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:34:40 UTC) orbiter warning: context deadline exceeded Duration: 23.299847982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XX..X...X.X.XXXXXXXXXXX.X...XX.X...X....XXX.X.X.X.XXX.X..X.] duration: 23.444951796s trace: https://jaeger.dev.storj.io/trace/c1c0c34ae9baaac FAIL (2025-12-17 14:34:30 UTC) orbiter warning: context deadline exceeded Duration: 23.444978928s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.XX.XXXXX] duration: 5.001718447s trace: https://jaeger.dev.storj.io/trace/3c337605b3458717 FAIL (2025-12-17 14:34:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001743364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.96884327s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:34:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.968881934s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X...X...............X........X.....X.XX....................] duration: 21.243315275s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:34:00 UTC) orbiter warning: context deadline exceeded Duration: 21.243340766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.XX.XXXX] duration: 5.000518371s trace: https://jaeger.dev.storj.io/trace/30ad4c8489b69106 FAIL (2025-12-17 14:33:59 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.000548158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.617877501s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:33:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.617900737s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [.XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.XX.XXX] duration: 5.001377979s trace: https://jaeger.dev.storj.io/trace/698c933a4718fc4 FAIL (2025-12-17 14:33:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001406834s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..XX..X...X.X.XXXXXXXXXXX.X...XX.X...X....XXX.X.X.X.XXX.X..X] duration: 30.000684202s trace: https://jaeger.dev.storj.io/trace/c1c0c34ae9baaac FAIL (2025-12-17 14:33:35 UTC) orbiter warning: context deadline exceeded Duration: 30.000697709s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.832696527s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:33:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.832726523s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X...X...............X........X.....X.XX...................] duration: 9.062295458s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:33:31 UTC) orbiter warning: context deadline exceeded Duration: 9.062321668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 11.626302083s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:33:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.626329004s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 11.50774001s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:33:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.507770999s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...X...X...............X........X.....X.XX..................] duration: 26.464030711s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:33:14 UTC) orbiter warning: context deadline exceeded Duration: 26.46404539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...XX..X...X.X.XXXXXXXXXXX.X...XX.X...X....XXX.X.X.X.XXX.X..] duration: 28.800880147s trace: https://jaeger.dev.storj.io/trace/1328062d983cdf71 FAIL (2025-12-17 14:33:05 UTC) orbiter warning: context deadline exceeded Duration: 28.800908822s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....X...X...............X........X.....X.XX.................] duration: 26.04001008s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:32:47 UTC) orbiter warning: context deadline exceeded Duration: 26.04005218s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [..XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.XX.XX] duration: 5.0005899s trace: https://jaeger.dev.storj.io/trace/10a33add0b9c2adf FAIL (2025-12-17 14:32:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000614916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.85137939s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:32:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.851407775s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 16.429062435s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:32:17 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 16.429090901s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.862539186s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:32:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.862563592s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [....XX..X...X.X.XXXXXXXXXXX.X...XX.X...X....XXX.X.X.X.XXX.X.] duration: 25.925163947s trace: https://jaeger.dev.storj.io/trace/1328062d983cdf71 FAIL (2025-12-17 14:31:55 UTC) orbiter warning: context deadline exceeded Duration: 25.925206137s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X..XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.XX.X] duration: 5.001872186s trace: https://jaeger.dev.storj.io/trace/28fb8a323b4e4743 FAIL (2025-12-17 14:31:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001894699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.772697104s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:31:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.772738161s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....X...X...............X........X.....X.XX................] duration: 20.820188974s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:31:38 UTC) orbiter warning: context deadline exceeded Duration: 20.820201599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X....XX..X...X.X.XXXXXXXXXXX.X...XX.X...X....XXX.X.X.X.XXX.X] duration: 30.001250929s trace: https://jaeger.dev.storj.io/trace/1328062d983cdf71 FAIL (2025-12-17 14:31:17 UTC) orbiter warning: context deadline exceeded Duration: 30.001281006s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......X...X...............X........X.....X.XX...............] duration: 21.752722694s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:31:17 UTC) orbiter warning: context deadline exceeded Duration: 21.752877789s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [.X..XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.XX.] duration: 4.188672909s trace: https://jaeger.dev.storj.io/trace/5c548a5dd95c63cd FAIL (2025-12-17 14:31:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.188698798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.781409305s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:31:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.781435114s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.97503017s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:31:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.975061s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.X..XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.XX] duration: 5.003631363s trace: https://jaeger.dev.storj.io/trace/5c548a5dd95c63cd FAIL (2025-12-17 14:30:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003657112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 8.001128615s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:30:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.001155556s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.......X...X...............X........X.....X.XX..............] duration: 30.753441482s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:30:14 UTC) orbiter warning: context deadline exceeded Duration: 30.753479535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X....XX..X...X.X.XXXXXXXXXXX.X...XX.X...X....XXX.X.X.X.XXX.] duration: 27.181878345s trace: https://jaeger.dev.storj.io/trace/1c36d06446e3c188 FAIL (2025-12-17 14:30:07 UTC) orbiter warning: context deadline exceeded Duration: 27.181909864s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XX.X..XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.X] duration: 5.00043543s trace: https://jaeger.dev.storj.io/trace/2c3ba2c50b9e03c3 FAIL (2025-12-17 14:30:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000533457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.217642586s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 6.217673917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.728976657s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:29:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.729007185s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 15.129736764s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:29:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 15.129763484s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.917164407s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:29:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.91719765s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [.XX.X..XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX.] duration: 3.863357839s trace: https://jaeger.dev.storj.io/trace/43954f17f0a35fe4 FAIL (2025-12-17 14:29:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.863383256s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........X...X...............X........X.....X.XX.............] duration: 18.95333431s trace: https://jaeger.dev.storj.io/trace/4beef8fb89704bac FAIL (2025-12-17 14:29:00 UTC) orbiter warning: context deadline exceeded Duration: 18.953378143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..X....XX..X...X.X.XXXXXXXXXXX.X...XX.X...X....XXX.X.X.X.XXX] duration: 30.001721888s trace: https://jaeger.dev.storj.io/trace/1c36d06446e3c188 FAIL (2025-12-17 14:28:56 UTC) orbiter warning: context deadline exceeded Duration: 30.001735875s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.143265246s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:28:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 6.143342994s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.XX.X..XXXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXXXX.X.XXXX] duration: 5.001232444s trace: https://jaeger.dev.storj.io/trace/43954f17f0a35fe4 FAIL (2025-12-17 14:28:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001259456s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)