Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................................X] duration: 30.001024943s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 14:58:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001037807s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XX] duration: 5.000988733s trace: https://jaeger.dev.storj.io/trace/4346e8b72f2ce037 FAIL (2025-12-17 14:58:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001024001s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X] duration: 30.001254811s trace: https://jaeger.dev.storj.io/trace/66150efdda4e0a18 FAIL (2025-12-17 14:58:05 UTC) orbiter warning: context deadline exceeded Duration: 30.001276031s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.057010118s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:58:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 6.057076194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.725040208s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:57:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.725070677s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..] duration: 30.000519952s trace: https://jaeger.dev.storj.io/trace/f33a856b0f7cfd FAIL (2025-12-17 14:57:25 UTC) orbiter warning: context deadline exceeded Duration: 30.0005456s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 12.301655778s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:57:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.301688771s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.946947239s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:57:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.946969961s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.X] duration: 5.001063396s trace: https://jaeger.dev.storj.io/trace/1ec757f4b785222e FAIL (2025-12-17 14:56:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001199686s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X.....X.XX...........................X....X..........X.X..] duration: 7.923548048s trace: https://jaeger.dev.storj.io/trace/44d596ce32f6088f FAIL (2025-12-17 14:56:50 UTC) orbiter warning: context deadline exceeded Duration: 7.923589216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X.] duration: 24.952764823s trace: https://jaeger.dev.storj.io/trace/f33a856b0f7cfd FAIL (2025-12-17 14:56:33 UTC) orbiter warning: context deadline exceeded Duration: 24.952798216s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.] duration: 3.656452478s trace: https://jaeger.dev.storj.io/trace/f22326dc7ee3880 FAIL (2025-12-17 14:56:19 UTC) orbiter warning: context deadline exceeded Duration: 3.656492684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 8.900964684s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:56:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.900991455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.771180276s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:56:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.771214271s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X] duration: 30.001617153s trace: https://jaeger.dev.storj.io/trace/f33a856b0f7cfd FAIL (2025-12-17 14:56:06 UTC) orbiter warning: context deadline exceeded Duration: 30.001640898s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X.....X.XX...........................X....X..........X.X.] duration: 30.00021338s trace: https://jaeger.dev.storj.io/trace/44d596ce32f6088f FAIL (2025-12-17 14:55:58 UTC) orbiter warning: context deadline exceeded Duration: 30.000231554s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.254846735s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:55:44 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.25487587s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX] duration: 5.000779236s trace: https://jaeger.dev.storj.io/trace/f22326dc7ee3880 FAIL (2025-12-17 14:55:32 UTC) orbiter warning: context deadline exceeded Duration: 5.000814974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....X.....X.XX...........................X....X..........X.X] duration: 1m0.001178409s trace: https://jaeger.dev.storj.io/trace/44d596ce32f6088f FAIL (2025-12-17 14:55:28 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001201783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.657391826s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:55:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.657417084s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.939601895s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:55:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.939618667s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.] duration: 25.65374248s trace: https://jaeger.dev.storj.io/trace/12ee80b9d5e59276 FAIL (2025-12-17 14:55:01 UTC) orbiter warning: context deadline exceeded Duration: 25.653774772s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XX] duration: 5.000651602s trace: https://jaeger.dev.storj.io/trace/5651709973b43c56 FAIL (2025-12-17 14:54:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000738207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 10.332319328s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:54:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.332346189s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.069847394s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:54:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 6.069878343s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.X] duration: 5.001590931s trace: https://jaeger.dev.storj.io/trace/77e845aa37dd6585 FAIL (2025-12-17 14:54:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00161633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X] duration: 30.001213601s trace: https://jaeger.dev.storj.io/trace/12ee80b9d5e59276 FAIL (2025-12-17 14:53:57 UTC) orbiter warning: context deadline exceeded Duration: 30.001236205s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....X.....X.XX...........................X....X..........X.] duration: 22.754736958s trace: https://jaeger.dev.storj.io/trace/26e4116343ec2138 FAIL (2025-12-17 14:53:54 UTC) orbiter warning: context deadline exceeded Duration: 22.754752686s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.997384158s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:53:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.997413162s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.] duration: 3.84602633s trace: https://jaeger.dev.storj.io/trace/11537638b332900b FAIL (2025-12-17 14:53:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.846057149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......X.....X.XX...........................X....X..........X] duration: 1m0.001974115s trace: https://jaeger.dev.storj.io/trace/26e4116343ec2138 FAIL (2025-12-17 14:53:31 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001995697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.778582952s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:53:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.77861361s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX] duration: 5.001283276s trace: https://jaeger.dev.storj.io/trace/11537638b332900b FAIL (2025-12-17 14:53:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001312802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...XX.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.] duration: 29.973303842s trace: https://jaeger.dev.storj.io/trace/2ec5c50a81a798d2 FAIL (2025-12-17 14:53:21 UTC) orbiter warning: context deadline exceeded Duration: 29.973321124s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.367254989s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:53:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.367285117s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XX] duration: 5.001328692s trace: https://jaeger.dev.storj.io/trace/19c7fab432b073d6 FAIL (2025-12-17 14:53:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00135312s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 8.257086708s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:53:02 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.257164085s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X...XX.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X] duration: 30.000974506s trace: https://jaeger.dev.storj.io/trace/2ec5c50a81a798d2 FAIL (2025-12-17 14:52:51 UTC) orbiter warning: context deadline exceeded Duration: 30.001000716s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...X] duration: 5.002515283s trace: https://jaeger.dev.storj.io/trace/38003e54af86fc71 FAIL (2025-12-17 14:52:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002540481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.791825709s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:52:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.791850917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.864913189s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:52:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.864938728s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 7.020559145s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:52:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.020594192s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X...XX.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..] duration: 27.758388501s trace: https://jaeger.dev.storj.io/trace/31b8f132b4b5c704 FAIL (2025-12-17 14:51:57 UTC) orbiter warning: context deadline exceeded Duration: 27.758491837s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......X.....X.XX...........................X....X..........] duration: 5.949039733s trace: https://jaeger.dev.storj.io/trace/3991b0e55a564d7a FAIL (2025-12-17 14:51:33 UTC) orbiter warning: context deadline exceeded Duration: 5.949071564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...] duration: 4.671346938s trace: https://jaeger.dev.storj.io/trace/193603abda39cb06 FAIL (2025-12-17 14:51:26 UTC) orbiter warning: context deadline exceeded Duration: 4.671378358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X...XX.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X.] duration: 23.023432827s trace: https://jaeger.dev.storj.io/trace/31b8f132b4b5c704 FAIL (2025-12-17 14:51:18 UTC) orbiter warning: context deadline exceeded Duration: 23.023505314s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.764422338s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 14:51:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.764455891s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.737641694s trace: https://jaeger.dev.storj.io/trace/3a4b0fa83b6b5dd9 FAIL (2025-12-17 14:50:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.737666651s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX.X...XX.X...X....XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X] duration: 30.001538313s trace: https://jaeger.dev.storj.io/trace/31b8f132b4b5c704 FAIL (2025-12-17 14:50:55 UTC) orbiter warning: context deadline exceeded Duration: 30.001560828s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........X.....X.XX...........................X....X.........] duration: 23.344017732s trace: https://jaeger.dev.storj.io/trace/3991b0e55a564d7a FAIL (2025-12-17 14:50:41 UTC) orbiter warning: context deadline exceeded Duration: 23.344050053s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)