Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX...X..X.] duration: 16.119482744s trace: https://jaeger.dev.storj.io/trace/4922d1d6f7597eac FAIL (2025-12-17 15:09:41 UTC) orbiter warning: context deadline exceeded Duration: 16.119509025s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXXXXXX..XX] duration: 5.001549342s trace: https://jaeger.dev.storj.io/trace/60f926195f7093d0 FAIL (2025-12-17 15:09:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001575833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.305025967s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:09:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.305062326s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX...X..X] duration: 30.001771567s trace: https://jaeger.dev.storj.io/trace/4922d1d6f7597eac FAIL (2025-12-17 15:09:05 UTC) orbiter warning: context deadline exceeded Duration: 30.001796315s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.978905818s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.978938741s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXXXXXX..X] duration: 5.00166424s trace: https://jaeger.dev.storj.io/trace/56252015d7f2b164 FAIL (2025-12-17 15:08:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001682695s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........................X....X..........X.X..X..............] duration: 6.89519684s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:08:37 UTC) orbiter warning: context deadline exceeded Duration: 6.895222507s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.........................X....X..........X.X..X.............] duration: 6.097041979s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:08:18 UTC) orbiter warning: context deadline exceeded Duration: 6.097080171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................X....X..........X.X..X............] duration: 19.626337989s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:08:12 UTC) orbiter warning: context deadline exceeded Duration: 19.626369369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.359097466s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:07:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.359131189s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX...X..] duration: 20.04892538s trace: https://jaeger.dev.storj.io/trace/58a0c1a1c2f571af FAIL (2025-12-17 15:07:53 UTC) orbiter warning: context deadline exceeded Duration: 20.048997838s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.8035425s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:07:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.803571024s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXXXXXX..] duration: 4.443287441s trace: https://jaeger.dev.storj.io/trace/1419e4f755d9519b FAIL (2025-12-17 15:07:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.443344921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXXXXXX.] duration: 4.1874334s trace: https://jaeger.dev.storj.io/trace/1419e4f755d9519b FAIL (2025-12-17 15:07:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.187460242s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXXXXXX] duration: 5.001496682s trace: https://jaeger.dev.storj.io/trace/1419e4f755d9519b FAIL (2025-12-17 15:07:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001538772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX...X.] duration: 22.687320405s trace: https://jaeger.dev.storj.io/trace/58a0c1a1c2f571af FAIL (2025-12-17 15:07:04 UTC) orbiter warning: context deadline exceeded Duration: 22.687357386s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........................X....X..........X.X..X...........] duration: 6.789503659s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:06:59 UTC) orbiter warning: context deadline exceeded Duration: 6.789533306s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.363844765s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 6.363876065s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 9.182278806s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:06:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.182308492s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX...X] duration: 30.00179071s trace: https://jaeger.dev.storj.io/trace/58a0c1a1c2f571af FAIL (2025-12-17 15:06:02 UTC) orbiter warning: context deadline exceeded Duration: 30.001815969s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXXXXX] duration: 5.001241726s trace: https://jaeger.dev.storj.io/trace/561c63372afa9ecd FAIL (2025-12-17 15:06:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001266002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X...........................X....X..........X.X..X..........] duration: 6.55980181s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:05:51 UTC) orbiter warning: context deadline exceeded Duration: 6.559833661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 11.34883186s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:05:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.348898318s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.15972311s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:05:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 7.15974931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XX...........................X....X..........X.X..X.........] duration: 21.131821586s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:05:36 UTC) orbiter warning: context deadline exceeded Duration: 21.131848847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.491479916s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:05:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.491558105s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.071057928s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:05:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 6.071088105s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXXXX] duration: 5.001144933s trace: https://jaeger.dev.storj.io/trace/7431b4a2d590b877 FAIL (2025-12-17 15:04:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001169309s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX...] duration: 18.415098713s trace: https://jaeger.dev.storj.io/trace/3faa55f00705b0f6 FAIL (2025-12-17 15:04:47 UTC) orbiter warning: context deadline exceeded Duration: 18.415140513s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 8.624703815s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:04:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.624724785s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.XX...........................X....X..........X.X..X........] duration: 17.375751452s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:04:41 UTC) orbiter warning: context deadline exceeded Duration: 17.37577655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.820494913s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:04:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.820530541s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX..] duration: 23.025418607s trace: https://jaeger.dev.storj.io/trace/3faa55f00705b0f6 FAIL (2025-12-17 15:04:26 UTC) orbiter warning: context deadline exceeded Duration: 23.025581155s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 11.399191034s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:04:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 11.399207475s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX.] duration: 23.748514084s trace: https://jaeger.dev.storj.io/trace/3faa55f00705b0f6 FAIL (2025-12-17 15:04:00 UTC) orbiter warning: context deadline exceeded Duration: 23.748531699s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXXX] duration: 5.002336569s trace: https://jaeger.dev.storj.io/trace/38c338d9ad5f5072 FAIL (2025-12-17 15:03:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002362749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.645672154s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:03:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.645704776s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XXX] duration: 30.00116657s trace: https://jaeger.dev.storj.io/trace/3faa55f00705b0f6 FAIL (2025-12-17 15:03:36 UTC) orbiter warning: context deadline exceeded Duration: 30.001192318s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 7.672137098s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:03:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.672166974s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.796255254s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:03:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.796281793s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X.XX...........................X....X..........X.X..X.......] duration: 11.930331619s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:03:24 UTC) orbiter warning: context deadline exceeded Duration: 11.930355886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXXX] duration: 5.001334522s trace: https://jaeger.dev.storj.io/trace/53f5fcc8b3ebc24c FAIL (2025-12-17 15:02:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001447538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.256641018s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:02:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.256666256s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..XX] duration: 30.001299927s trace: https://jaeger.dev.storj.io/trace/7047c74539e23dce FAIL (2025-12-17 15:02:36 UTC) orbiter warning: context deadline exceeded Duration: 30.001313112s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.846719226s trace: https://jaeger.dev.storj.io/trace/269f3a09cefa5304 FAIL (2025-12-17 15:02:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA) context deadline exceeded ======LOGS====== Dec 17 14:23:59.875948279 (286h54m59.747031314s) [upload=251555]: Starting upload Dec 17 14:23:59.876005657 (286h54m59.747088692s) [upload=251555]: Done starting upload Dec 17 14:23:59.876197153 (286h54m59.747280198s) [upload=251555]: Uploading segments... Dec 17 14:23:59.876299007 (286h54m59.747382052s) [upload=251555]: Got next segment. Inline: false Dec 17 14:23:59.876339674 (286h54m59.747422719s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 14:23:59.876346406 (286h54m59.747429452s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 14:24:00.124613924 (286h54m59.995696989s) [seg_pos={0 0}, upload=251555]: Begin upload segment... Dec 17 14:24:00.124676081 (286h54m59.995759126s) [seg_pos={0 0}, upload=251555]: Done begin upload segment. Dec 17 14:24:00.124715056 (286h54m59.995798102s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.124826919 (286h54m59.995909964s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125375444 (286h54m59.996458489s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125451969 (286h54m59.996535005s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125707846 (286h54m59.996790891s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.125863774 (286h54m59.996946819s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126111014 (286h54m59.99719406s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126201967 (286h54m59.997285012s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126456553 (286h54m59.997539588s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126581640 (286h54m59.997664685s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.126756253 (286h54m59.997839288s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127169409 (286h54m59.998252455s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127195309 (286h54m59.998278354s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127489889 (286h54m59.998572925s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.127604368 (286h54m59.998687403s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:00.128446622 (286h54m59.999529657s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Uploading piece... Dec 17 14:24:01.745844898 (286h55m1.616927943s) [upload=251555]: Next returned nil segment Dec 17 14:24:01.745865247 (286h55m1.616948292s) [upload=251555]: Waiting for error group managing segments... Dec 17 14:24:02.125399239 (286h55m1.996482284s) [noise=true , address=145.79.227.36:4518 , piece_id=GXBP4F3BZ2ZAFU7L6F2ORN6V432VBRVNBYQN3GLWDH727HSB7NIA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.126481550 (286h55m1.997564605s) [noise=true , address=145.79.227.25:4617 , piece_id=76BUQX3JFTWRCFO77QSN36X7GI5UFDDBCOWO2PUNGMTB7XW7XJEA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.127574442 (286h55m1.998657487s) [noise=true , address=145.79.227.8:4330 , piece_id=7VA57PV37XE2PLTLZSTFXAL3DWIGUPCLUDGLLNZFWF2L2C7QIOUQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144487878 (286h55m2.015570923s) [noise=true , address=145.79.227.48:4301 , piece_id=NWFW5XTFWAGTRK2XHCRHP4VS7RLQ2KMWFOPPL3OR2ZLBU4YU736A, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.144919459 (286h55m2.016002504s) [noise=true , address=145.79.227.31:4799 , piece_id=UEN4CLHW576UIF425PEM4K5ZX3E3HQYLOY55M72ILEIZBTI4GBBQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.206563161 (286h55m2.077646206s) [noise=true , address=145.79.227.37:4565 , piece_id=T6TKEWR2AJ4DTQ4EUMZMYQXQ7TTHBCREXJDDUDWTZYP7YHL23EKA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.220071653 (286h55m2.091154698s) [noise=true , address=145.79.227.43:4157 , piece_id=QL4JPEWPCU3OQZGYTUTIOYZ2PMIQCTA4CFEP7VNAQL77DUJYKJWQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.223129416 (286h55m2.094212461s) [noise=true , address=145.79.227.19:4077 , piece_id=D6BURLEYHSFWR3SFC4WQEA23MW22HKKXIBCWUM42HAWF4IAL6TRQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232568790 (286h55m2.103651865s) [noise=true , address=145.79.227.6:4258 , piece_id=ZUPRHDSYFW5Y7QW2L3MBMHYN4HSHIM6EUAK4LS3VPJGSGFMFR2AQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.232747801 (286h55m2.103830846s) [noise=true , address=145.79.227.41:4087 , piece_id=TYT6PNIZCSFFBEPH4ZO56BZ655ZKXBB57EHORC6Q3XVMDHA5TGJQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.233412857 (286h55m2.104495902s) [noise=true , address=145.79.227.39:4015 , piece_id=INDYRD3AU6DM5MB2AZLBYR5UCGUDMJLSLSAWDOO24GVFDFVGVJKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.402580213 (286h55m2.273663248s) [noise=true , address=45.91.116.8:4048 , piece_id=4V3JN6SRF23AHT7P4ZWBK2WIGDC7NFCT37UTSII3M2IJEXJXJMRA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407473961 (286h55m2.278557026s) [noise=true , address=45.91.116.7:4011 , piece_id=JOFAO3BWRKAMKOKIJLUHASUHUWF72AWCBNO3UQZ6KMPAGC7QQK6Q, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.407622134 (286h55m2.278705179s) [seg_pos={0 0}, upload=251555]: Segment reached optimal threshold of 13 pieces. Dec 17 14:24:02.407935050 (286h55m2.279018095s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.407948014 (286h55m2.279031059s) [noise=true , address=77.90.31.16:4736 , piece_id=76FWJER4KP2C7AQCPIHSXR5GTKKPWIJFIVUWBGYX3YOBCLF6TEKQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408286699 (286h55m2.279369744s) [noise=true , address=45.91.116.9:4263 , piece_id=YF6RQ5CVDKT6K4FHI7N34TBB2O5NW2CEIEQMAJ7WHV5EHH542ETA, seg_pos={0 0}, upload=251555]: Done uploading piece. err: <nil> Dec 17 14:24:02.408326715 (286h55m2.279409761s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 17 14:24:02.408338097 (286h55m2.279421142s) [noise=true , address=45.91.116.11:4120 , piece_id=XGJXD6MDPKR6HZJCGCI7TX5CLPITCGLQEDQRKMC3GDST53AFWTOQ, seg_pos={0 0}, upload=251555]: Long tail cancellation due to optimal threshold Dec 17 14:24:02.408367493 (286h55m2.279450538s) [seg_pos={0 0}, upload=251555]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 17 14:24:02.408459939 (286h55m2.279542984s) [upload=251555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 14:24:02.408466732 (286h55m2.279549777s) [upload=251555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 14:24:02.694441231 (286h55m2.565524286s) [upload=251555]: Done uploading segments... Dec 17 14:24:02.694468864 (286h55m2.565551899s) [upload=251555]: Upload finished. err: <nil> Duration: 5.846743651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XX.X.XXXX.XX.XXXXX.XXXXXXX..XXXX.XXX.XX.X...XXX.XXX.XXXXXXXX] duration: 5.001005657s trace: https://jaeger.dev.storj.io/trace/7598f72dabdbe701 FAIL (2025-12-17 15:02:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00107528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X.XX...........................X....X..........X.X..X......] duration: 7.405157923s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:02:15 UTC) orbiter warning: context deadline exceeded Duration: 7.405191045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...XXX.X.X.X.XXX.X..X.X.XX.X.X..X.XX.XX....X..X.X.X..X.XX..X] duration: 30.001363355s trace: https://jaeger.dev.storj.io/trace/6bd2482a4054e5ec FAIL (2025-12-17 15:02:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001390026s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X.XX...........................X....X..........X.X..X.....] duration: 19.753026595s trace: https://jaeger.dev.storj.io/trace/799536cea9266ecf FAIL (2025-12-17 15:02:00 UTC) orbiter warning: context deadline exceeded Duration: 19.753056562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 11.655423307s trace: https://jaeger.dev.storj.io/trace/efe666d0c62062e FAIL (2025-12-17 15:01:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.655453753s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)