Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 16.243212979s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:12:02 UTC) orbiter warning: context deadline exceeded Duration: 16.24324581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......X..............................X.........] duration: 14.280501048s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:11:55 UTC) orbiter warning: context deadline exceeded Duration: 14.280529735s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXXXXX.XXX.] duration: 4.451091965s trace: https://jaeger.dev.storj.io/trace/1fc96d9b95ffa6b4 FAIL (2025-12-17 16:11:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.45116327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............................X..........X.............] duration: 7.063029779s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:11:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 7.063061039s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......X..............................X........] duration: 17.564917783s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:11:32 UTC) orbiter warning: context deadline exceeded Duration: 17.56494765s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.366093581s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:11:17 UTC) orbiter warning: context deadline exceeded Duration: 6.366119118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X............X.......X..............................X.......] duration: 20.401656363s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:11:13 UTC) orbiter warning: context deadline exceeded Duration: 20.401684145s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXXXXX.XXX] duration: 5.000766335s trace: https://jaeger.dev.storj.io/trace/1fc96d9b95ffa6b4 FAIL (2025-12-17 16:10:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000791504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.............................X..........X............] duration: 5.800938483s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:10:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 5.800965263s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 12.895315322s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:10:27 UTC) orbiter warning: context deadline exceeded Duration: 12.895346812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX............X.......X..............................X......] duration: 15.322756558s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:10:23 UTC) orbiter warning: context deadline exceeded Duration: 15.322790744s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXXXXX.XX] duration: 5.001760001s trace: https://jaeger.dev.storj.io/trace/176eb19047243d93 FAIL (2025-12-17 16:10:07 UTC) orbiter warning: context deadline exceeded Duration: 5.001776954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.............................X..........X...........] duration: 5.899518345s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:10:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 5.899550617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.............................X..........X..........] duration: 6.251053392s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:09:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 6.251082589s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.045306355s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:09:15 UTC) orbiter warning: context deadline exceeded Duration: 6.045330952s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXX............X.......X..............................X.....] duration: 6.859214425s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:09:08 UTC) orbiter warning: context deadline exceeded Duration: 6.859245171s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [..X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXXXXX.X] duration: 5.001429067s trace: https://jaeger.dev.storj.io/trace/e069e84dc51f31c FAIL (2025-12-17 16:08:57 UTC) orbiter warning: context deadline exceeded Duration: 5.001450649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXX............X.......X..............................X....] duration: 12.975449878s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:08:42 UTC) orbiter warning: context deadline exceeded Duration: 12.975473663s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.............................X..........X.........] duration: 5.862353957s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:08:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 5.862382461s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.429425937s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:08:26 UTC) orbiter warning: context deadline exceeded Duration: 6.429453511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X..X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXXXXX.] duration: 3.598440928s trace: https://jaeger.dev.storj.io/trace/2b1d4cee3c3b4c83 FAIL (2025-12-17 16:08:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.59858864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.105891417s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:07:47 UTC) orbiter warning: context deadline exceeded Duration: 6.105924161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XX..X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXXXXX] duration: 5.001027936s trace: https://jaeger.dev.storj.io/trace/2b1d4cee3c3b4c83 FAIL (2025-12-17 16:07:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001040109s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXX............X.......X..............................X...] duration: 7.427964203s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:07:28 UTC) orbiter warning: context deadline exceeded Duration: 7.427991004s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.........X.............................X..........X........] duration: 8.55002331s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:07:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 8.550050713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.........X.............................X..........X.......] duration: 5.95589619s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:06:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 5.955927148s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 13.251053108s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:06:48 UTC) orbiter warning: context deadline exceeded Duration: 13.251113553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXX..X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXXXX] duration: 5.001535698s trace: https://jaeger.dev.storj.io/trace/4846bbc6d4067029 FAIL (2025-12-17 16:06:42 UTC) orbiter warning: context deadline exceeded Duration: 5.001555567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XXX............X.......X..............................X..] duration: 18.491977579s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:06:33 UTC) orbiter warning: context deadline exceeded Duration: 18.491995484s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.........X.............................X..........X......] duration: 5.785260747s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:06:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 5.785292167s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XXX............X.......X..............................X.] duration: 18.420677073s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:06:15 UTC) orbiter warning: context deadline exceeded Duration: 18.420690518s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.571166168s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:06:03 UTC) orbiter warning: context deadline exceeded Duration: 6.571200874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.........X.............................X..........X.....] duration: 6.060661902s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:05:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 6.06069214s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XXX............X.......X..............................X] duration: 30.000875907s trace: https://jaeger.dev.storj.io/trace/989e052fee10c42 FAIL (2025-12-17 16:05:56 UTC) orbiter warning: context deadline exceeded Duration: 30.000910463s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXX..X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXXX] duration: 5.001149083s trace: https://jaeger.dev.storj.io/trace/f6745f4ae8252ec FAIL (2025-12-17 16:05:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001169212s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 17.244434139s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:05:27 UTC) orbiter warning: context deadline exceeded Duration: 17.244464276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XXX............X.......X..............................] duration: 17.094616569s trace: https://jaeger.dev.storj.io/trace/421601e3c94c18c6 FAIL (2025-12-17 16:05:10 UTC) orbiter warning: context deadline exceeded Duration: 17.094645425s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.........X.............................X..........X....] duration: 6.111145254s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:04:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 6.111174229s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 13.279587218s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:04:48 UTC) orbiter warning: context deadline exceeded Duration: 13.279605974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXXX..X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXXX] duration: 5.000871267s trace: https://jaeger.dev.storj.io/trace/a7ccae70a6cb2b0 FAIL (2025-12-17 16:04:41 UTC) orbiter warning: context deadline exceeded Duration: 5.000897026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XXX............X.......X.............................] duration: 19.114474018s trace: https://jaeger.dev.storj.io/trace/421601e3c94c18c6 FAIL (2025-12-17 16:04:33 UTC) orbiter warning: context deadline exceeded Duration: 19.114500408s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [.XXXXX..X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XXX] duration: 5.000944373s trace: https://jaeger.dev.storj.io/trace/3e6b2a9087c060f7 FAIL (2025-12-17 16:04:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001035125s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.229535701s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:03:48 UTC) orbiter warning: context deadline exceeded Duration: 6.22956181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.........X.............................X..........X...] duration: 5.915031566s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:03:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 5.91506034s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 13.757866035s trace: https://jaeger.dev.storj.io/trace/5e15c2a276fabadf FAIL (2025-12-17 16:03:31 UTC) orbiter warning: context deadline exceeded Duration: 13.757896153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.........X.............................X..........X..] duration: 6.004345134s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:03:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 6.004371546s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......XXX............X.......X............................] duration: 7.392396351s trace: https://jaeger.dev.storj.io/trace/421601e3c94c18c6 FAIL (2025-12-17 16:03:15 UTC) orbiter warning: context deadline exceeded Duration: 7.392445986s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X.XXXXX..X....XXX..XXXX.X.XXX..XX.X..XXXXX..X.X...XXXX.X..XX] duration: 5.001751781s trace: https://jaeger.dev.storj.io/trace/51af04a2485bbdc3 FAIL (2025-12-17 16:03:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001778062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.........X.............................X..........X.] duration: 5.880360897s trace: https://jaeger.dev.storj.io/trace/34e320dfdcd6aecd FAIL (2025-12-17 16:02:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 16:01:57.930656800 (288h32m57.801739845s) [upload=252966]: Starting upload Dec 17 16:01:57.930721494 (288h32m57.801804529s) [upload=252966]: Done starting upload Dec 17 16:01:57.930799873 (288h32m57.801882908s) [upload=252966]: Uploading segments... Dec 17 16:01:57.930918299 (288h32m57.802001334s) [upload=252966]: Got next segment. Inline: false Dec 17 16:01:57.930949929 (288h32m57.802032964s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 16:01:57.930954067 (288h32m57.802037102s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 16:01:58.246439218 (288h32m58.117522253s) [seg_pos={0 0}, upload=252966]: Begin upload segment... Dec 17 16:01:58.246487160 (288h32m58.117570205s) [seg_pos={0 0}, upload=252966]: Done begin upload segment. Dec 17 16:01:58.246513150 (288h32m58.117596195s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.246543877 (288h32m58.117626923s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247068117 (288h32m58.118151152s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247420688 (288h32m58.118503733s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.247869963 (288h32m58.118953009s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248293319 (288h32m58.119376365s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.248686999 (288h32m58.119770034s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249143338 (288h32m58.120226383s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249480250 (288h32m58.120563295s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249614255 (288h32m58.12069729s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.249842018 (288h32m58.120925053s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250112393 (288h32m58.121195428s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250126069 (288h32m58.121209114s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250381586 (288h32m58.121464621s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250640799 (288h32m58.121723834s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:58.250928216 (288h32m58.122011251s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Uploading piece... Dec 17 16:01:59.956077117 (288h32m59.827160162s) [upload=252966]: Next returned nil segment Dec 17 16:01:59.956091413 (288h32m59.827174458s) [upload=252966]: Waiting for error group managing segments... Dec 17 16:02:00.234684707 (288h33m0.105767742s) [noise=true , address=145.79.227.39:4014 , piece_id=NDHHB52SRXSA43P2XKANPJ7DPXHREWEDE3BL5LLHNP6SS2RAXTLA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.241492359 (288h33m0.112575394s) [noise=true , address=145.79.227.31:4797 , piece_id=7AW4WJJHRUAX5645XB4IQMEANVMVDUGBOO5U6RSJOIFDOIS6PUOA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.253195615 (288h33m0.12427865s) [noise=true , address=145.79.227.49:4339 , piece_id=T3UASJUMS2COCHIFFFR6PIZGITWET4RY6Z2EUTRVYIZ3EOPVGBSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.293461429 (288h33m0.164544484s) [noise=true , address=145.79.227.50:4373 , piece_id=QVAH3NDAPV3X3GYCUNZHJXFWQY34WLCR5YPPGMOCSXY5M6L5KNUQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.336648294 (288h33m0.207731359s) [noise=true , address=145.79.227.52:4445 , piece_id=6YBRUG6DYF2SLMEF2LNQHFVIQ75NLQOPZJKLUXP2KVXCSTZCNAXQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.444497830 (288h33m0.315580865s) [noise=true , address=77.90.31.12:4590 , piece_id=NQCYFYLEUBQXNGEKPOMNPCCSS5NC2MN7KA6KLWVZYERWBM6HQEEQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:00.472673461 (288h33m0.343756506s) [noise=true , address=77.90.31.18:4808 , piece_id=ZOTBQ6UXREQPKNZ2QR7NU5D776WLSFSSJXAAFJEJ7OVP64FCM6PQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.245188333 (288h33m1.116271388s) [noise=true , address=77.90.31.11:4555 , piece_id=O7XD7GBQCRMAI2IFMETMVGMU4DIR3QNGHKKCAC54TU2Q7SOU4LYQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:01.994712535 (288h33m1.86579558s) [noise=true , address=145.79.227.48:4301 , piece_id=VPU5R7YWK7OZECUDI7W67XXQLFUZASNCWW3YOI4U723M5SZZOEMQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.258006075 (288h33m2.12908912s) [noise=true , address=45.91.116.13:4156 , piece_id=LXU6DHWF5LDIUHEA3NU2AOUK43NBH7JG4JW4JUKYGOQNR7W22BSA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:02.561672401 (288h33m2.432755446s) [noise=true , address=145.79.227.34:4906 , piece_id=7JD4K37RAQNS2W4LDQ6G5UROZ6QQNJN4MSGOJLXEUICMTR4ZHUYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.034114689 (288h33m2.905197754s) [noise=true , address=145.79.227.43:4157 , piece_id=L77TS3KVPEWIUWLJ3MVDT3JVVGGSFRE2UY2AJRYGFQHRV6PNPWSQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713021353 (288h33m3.584104398s) [noise=true , address=45.91.116.7:4010 , piece_id=62JBCENP7IX6GVRPMPS5644WT3BKVBBOV3ZPHZ5KJAEU6L536T5A, seg_pos={0 0}, upload=252966]: Done uploading piece. err: <nil> Dec 17 16:02:03.713048946 (288h33m3.584131991s) [seg_pos={0 0}, upload=252966]: Segment reached optimal threshold of 13 pieces. Dec 17 16:02:03.713377932 (288h33m3.584460977s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713388462 (288h33m3.584471528s) [noise=true , address=145.79.227.42:4122 , piece_id=VSCF2VFFJSP2NYQAULKMJQAU5HNRXDDU7ENCC7ZZ2MFHIHT6UISA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.713911909 (288h33m3.584994954s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.713922790 (288h33m3.585005835s) [noise=true , address=45.91.116.14:4192 , piece_id=I7EPKLMCUY7YIWK7OYD3BZPTCXTE5NINQEQCBOZDCJDP5NBVAHYA, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714362387 (288h33m3.585445432s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 17 16:02:03.714373027 (288h33m3.585456092s) [noise=true , address=45.91.116.11:4119 , piece_id=TIWOZR6PM2BBYNP5HAPQQ6IZGSMGJ4AMOWFAXBLKU33BDREG2ROQ, seg_pos={0 0}, upload=252966]: Long tail cancellation due to optimal threshold Dec 17 16:02:03.714444753 (288h33m3.585527799s) [seg_pos={0 0}, upload=252966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 16:02:03.714683689 (288h33m3.585766734s) [upload=252966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 16:02:03.714692275 (288h33m3.58577532s) [upload=252966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 16:02:04.099782398 (288h33m3.970865443s) [upload=252966]: Done uploading segments... Dec 17 16:02:04.099805502 (288h33m3.970888547s) [upload=252966]: Upload finished. err: <nil> Duration: 5.880387558s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.......XXX............X.......X...........................] duration: 17.388243861s trace: https://jaeger.dev.storj.io/trace/421601e3c94c18c6 FAIL (2025-12-17 16:02:45 UTC) orbiter warning: context deadline exceeded Duration: 17.388275341s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)