Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X.......] duration: 3.626622856s trace: https://jaeger.dev.storj.io/trace/455cace4d9600a91 FAIL (2025-12-15 08:33:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.626650158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X......X..............] duration: 7.045716237s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:33:20 UTC) orbiter warning: context deadline exceeded Duration: 7.045788295s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X......] duration: 3.585392738s trace: https://jaeger.dev.storj.io/trace/455cace4d9600a91 FAIL (2025-12-15 08:33:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.585416212s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 8.770157311s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:32:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 8.770177558s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......X......X.............] duration: 19.918001407s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:32:31 UTC) orbiter warning: context deadline exceeded Duration: 19.918032896s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X.....] duration: 3.617408002s trace: https://jaeger.dev.storj.io/trace/455cace4d9600a91 FAIL (2025-12-15 08:32:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.617440725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 10.910901957s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:31:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 10.910948966s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X....] duration: 3.539999618s trace: https://jaeger.dev.storj.io/trace/455cace4d9600a91 FAIL (2025-12-15 08:31:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540026751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.......X......X............] duration: 11.986526921s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:31:28 UTC) orbiter warning: context deadline exceeded Duration: 11.986551818s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 9.16750362s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:31:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 9.167525873s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.......X......X...........] duration: 18.359310204s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:31:02 UTC) orbiter warning: context deadline exceeded Duration: 18.361119701s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X...] duration: 3.642686016s trace: https://jaeger.dev.storj.io/trace/455cace4d9600a91 FAIL (2025-12-15 08:30:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.642734519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X..] duration: 3.61096486s trace: https://jaeger.dev.storj.io/trace/455cace4d9600a91 FAIL (2025-12-15 08:30:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611035654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 8.907164962s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 8.907197665s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.......X......X..........] duration: 14.85564301s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:30:28 UTC) orbiter warning: context deadline exceeded Duration: 14.855661817s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.......X......X.........] duration: 20.550404499s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:30:12 UTC) orbiter warning: context deadline exceeded Duration: 20.550427064s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 8.618250722s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:29:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 8.618276601s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.......X......X........] duration: 29.925532742s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:29:52 UTC) orbiter warning: context deadline exceeded Duration: 29.925596834s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X.] duration: 3.637553634s trace: https://jaeger.dev.storj.io/trace/455cace4d9600a91 FAIL (2025-12-15 08:29:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.637570536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 8.422977124s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:29:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 8.423024273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 9.59274267s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:28:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 9.592766104s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.......X......X.......] duration: 7.112609103s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:28:17 UTC) orbiter warning: context deadline exceeded Duration: 7.112641044s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 9.523400039s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:28:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 9.523431999s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.X] duration: 5.00199322s trace: https://jaeger.dev.storj.io/trace/455cace4d9600a91 FAIL (2025-12-15 08:28:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002041443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.......X......X......] duration: 23.659381772s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:27:51 UTC) orbiter warning: context deadline exceeded Duration: 23.659432739s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 8.340306128s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:27:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 8.340351154s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X.] duration: 3.67748495s trace: https://jaeger.dev.storj.io/trace/1ed97b107ca652c FAIL (2025-12-15 08:27:06 UTC) orbiter warning: context deadline exceeded Duration: 3.677509255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.......X......X.....] duration: 25.559407554s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:26:57 UTC) orbiter warning: context deadline exceeded Duration: 25.559439083s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 8.930145797s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:26:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 8.930176295s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [....XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...X] duration: 5.001820845s trace: https://jaeger.dev.storj.io/trace/1ed97b107ca652c FAIL (2025-12-15 08:26:46 UTC) orbiter warning: context deadline exceeded Duration: 5.001841624s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.....XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX...] duration: 3.467421699s trace: https://jaeger.dev.storj.io/trace/25d5a731f74f4d99 FAIL (2025-12-15 08:26:23 UTC) orbiter warning: context deadline exceeded Duration: 3.467446355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003967743s trace: https://jaeger.dev.storj.io/trace/27df128ac06fc9a5 FAIL (2025-12-15 08:26:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 08:25:52.083445632 (232h56m51.954528677s) [upload=204259]: Starting upload Dec 15 08:25:52.083510746 (232h56m51.954593781s) [upload=204259]: Done starting upload Dec 15 08:25:52.083667675 (232h56m51.95475073s) [upload=204259]: Uploading segments... Dec 15 08:25:52.084449764 (232h56m51.955532799s) [upload=204259]: Got next segment. Inline: false Dec 15 08:25:52.084503136 (232h56m51.955586181s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 08:25:52.084508456 (232h56m51.955591491s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 08:25:52.410453161 (232h56m52.281536206s) [seg_pos={0 0}, upload=204259]: Begin upload segment... Dec 15 08:25:52.410572447 (232h56m52.281655493s) [seg_pos={0 0}, upload=204259]: Done begin upload segment. Dec 15 08:25:52.410677888 (232h56m52.281761204s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.411751292 (232h56m52.282834337s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412136215 (232h56m52.28321925s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412584299 (232h56m52.283667344s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.412957680 (232h56m52.284040725s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413347432 (232h56m52.284430477s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.413754427 (232h56m52.284837472s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414278946 (232h56m52.285361991s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.414809466 (232h56m52.285892511s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415241429 (232h56m52.286324474s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.415887248 (232h56m52.286970303s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416289084 (232h56m52.287372129s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416631536 (232h56m52.287714581s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.416922680 (232h56m52.288005725s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417201220 (232h56m52.288284265s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:52.417667619 (232h56m52.288750664s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Uploading piece... Dec 15 08:25:54.309303638 (232h56m54.180386683s) [upload=204259]: Next returned nil segment Dec 15 08:25:54.309319818 (232h56m54.180402873s) [upload=204259]: Waiting for error group managing segments... Dec 15 08:25:54.673098568 (232h56m54.544181603s) [noise=true , address=45.91.116.8:4045 , piece_id=63JDLRSN325CWMYJJ3YY3D2F36B7WCNATUA6LBT7LPIXXL4CD5ZQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.778330160 (232h56m54.649413195s) [noise=true , address=77.90.31.11:4555 , piece_id=LY7IYOOBMJZTAPS2GFOO2KPFA5ADWTOHUPWQCR4PTRDSQP6V653Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.932879913 (232h56m54.803962948s) [noise=true , address=145.79.227.52:4445 , piece_id=2TXRX5SLKO2LHEKG2IKHF6J3QZWC2XLQO26XVK574ZKXKPK5DLGA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938549900 (232h56m54.809632945s) [noise=true , address=145.79.227.50:4375 , piece_id=FLZAWDVQURSMZHFNBRKKKROVXPC4XJTMN5IFDW2AVJLANCJK3M3Q, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.938745002 (232h56m54.809828047s) [noise=true , address=145.79.227.45:4231 , piece_id=J4OFBPM2K5YLTGT56HFIO3WSAQQ3ODEUQKADLP67KZTLR6O4JFQA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.939390661 (232h56m54.810473696s) [noise=true , address=145.79.227.10:4401 , piece_id=O3YBSSUZ7KNXTQXMRLJZPWIQJMCPA4OWKJEVGHUN24PW7KOQQY4A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.941037067 (232h56m54.812120102s) [noise=true , address=145.79.227.46:4603 , piece_id=V6KICMXQRA4PXIJV4CBDRNB7BYV7SVPKOGL4IHE4QE4CAULR7Q2A, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.945744130 (232h56m54.816827165s) [noise=true , address=145.79.227.5:4223 , piece_id=7VAFHJBBLNQEBPIGNKDR2QYLJMTQXW5KJTKNZWJHCZCCP6V552TA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:54.955787875 (232h56m54.82687092s) [noise=true , address=145.79.227.47:4265 , piece_id=W3DFALE65JACCM5TTY72WVXYP67MQ5TOBQGXYFRCFYIA5JZBS2XA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.143188213 (232h56m55.014271258s) [noise=true , address=145.79.227.9:4367 , piece_id=BOQYAZ3DENGYGRNBSRG6LWQPFOCV23EBRWBCUZSBABSUI5L7UNFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.158818357 (232h56m55.029901402s) [noise=true , address=145.79.227.50:4374 , piece_id=AS7FSQO6SEZPQRWN7D4SIF36GVH65N36ZQLXER5NQ7ZEBYWUUNMQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.159916487 (232h56m55.030999532s) [noise=true , address=145.79.227.32:4835 , piece_id=2XU3QHVQG4XHBM4R5V62SKXNFNBSNVLRIGKD6GNB2BZW7OUKA5RA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160952270 (232h56m55.032035325s) [noise=true , address=145.79.227.34:4907 , piece_id=X3HK4W4EUKE4Y44CZHK7FFWUFKZQ6JMZAU24TM6VUWMSQNSHMVYA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: <nil> Dec 15 08:25:55.160970405 (232h56m55.03205348s) [seg_pos={0 0}, upload=204259]: Segment reached optimal threshold of 13 pieces. Dec 15 08:25:55.161222154 (232h56m55.032305189s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161232042 (232h56m55.032315078s) [noise=true , address=145.79.227.26:4654 , piece_id=WRSBZKH6DBWJX57CFSGTDWNHTQAZC7MKAUHEJSND27M725DM6MFA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161533878 (232h56m55.032616913s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161543075 (232h56m55.03262612s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZDNOYOMDXFYGTL6WQICYAWN7FH5OX4Q5JJ3HOMDDSOG7YMCOXQ, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161702268 (232h56m55.032785303s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 15 08:25:55.161710454 (232h56m55.032793499s) [noise=true , address=145.79.227.20:4438 , piece_id=5RX3TQ2H447MUCZ52L3BSNQZN5VTBG6RMUPYQKXUXJ2GULOFIJAA, seg_pos={0 0}, upload=204259]: Long tail cancellation due to optimal threshold Dec 15 08:25:55.161731935 (232h56m55.03281498s) [seg_pos={0 0}, upload=204259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 08:25:55.161871300 (232h56m55.032954345s) [upload=204259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 08:25:55.161879366 (232h56m55.032962411s) [upload=204259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 08:25:55.517689320 (232h56m55.388772365s) [upload=204259]: Done uploading segments... Dec 15 08:25:55.517709168 (232h56m55.388792203s) [upload=204259]: Upload finished. err: <nil> Duration: 30.003988551s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.......X......X....] duration: 26.938227014s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:26:05 UTC) orbiter warning: context deadline exceeded Duration: 26.938254526s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [......XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX..] duration: 3.357514337s trace: https://jaeger.dev.storj.io/trace/25d5a731f74f4d99 FAIL (2025-12-15 08:25:52 UTC) orbiter warning: context deadline exceeded Duration: 3.357535076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.......XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX.] duration: 3.606229043s trace: https://jaeger.dev.storj.io/trace/25d5a731f74f4d99 FAIL (2025-12-15 08:25:37 UTC) orbiter warning: context deadline exceeded Duration: 3.606253539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.......X......X...] duration: 7.555752119s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:25:24 UTC) orbiter warning: context deadline exceeded Duration: 7.555783078s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [........XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XXX] duration: 5.001419198s trace: https://jaeger.dev.storj.io/trace/25d5a731f74f4d99 FAIL (2025-12-15 08:25:09 UTC) orbiter warning: context deadline exceeded Duration: 5.001443364s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.......X......X..] duration: 20.770411542s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:25:06 UTC) orbiter warning: context deadline exceeded Duration: 20.770439405s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.......X......X.] duration: 19.891325284s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:24:20 UTC) orbiter warning: context deadline exceeded Duration: 19.89133914s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.........XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.XX] duration: 5.001263635s trace: https://jaeger.dev.storj.io/trace/677c85095de4eb6b FAIL (2025-12-15 08:24:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001289995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.......X......X] duration: 30.001908976s trace: https://jaeger.dev.storj.io/trace/27be0b03abe9c003 FAIL (2025-12-15 08:24:00 UTC) orbiter warning: context deadline exceeded Duration: 30.001935796s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.........XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.X] duration: 5.001020048s trace: https://jaeger.dev.storj.io/trace/30f7736c055a81b3 FAIL (2025-12-15 08:23:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001054082s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.......X......] duration: 7.297367034s trace: https://jaeger.dev.storj.io/trace/5d3d0371eac0fbf2 FAIL (2025-12-15 08:22:24 UTC) orbiter warning: context deadline exceeded Duration: 7.297387884s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X.........XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X.] duration: 3.538522414s trace: https://jaeger.dev.storj.io/trace/67628583421f9234 FAIL (2025-12-15 08:22:17 UTC) orbiter warning: context deadline exceeded Duration: 3.538548895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.......X.....] duration: 20.936787192s trace: https://jaeger.dev.storj.io/trace/5d3d0371eac0fbf2 FAIL (2025-12-15 08:21:56 UTC) orbiter warning: context deadline exceeded Duration: 20.936811389s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.......X....] duration: 7.362477216s trace: https://jaeger.dev.storj.io/trace/5d3d0371eac0fbf2 FAIL (2025-12-15 08:21:27 UTC) orbiter warning: context deadline exceeded Duration: 7.362509608s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..X.........XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.X] duration: 5.001170424s trace: https://jaeger.dev.storj.io/trace/67628583421f9234 FAIL (2025-12-15 08:21:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001208928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.......X...] duration: 19.550333951s trace: https://jaeger.dev.storj.io/trace/5d3d0371eac0fbf2 FAIL (2025-12-15 08:20:36 UTC) orbiter warning: context deadline exceeded Duration: 19.55036987s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...X.........XX.X.....X..X.....X...X...X...X.....XXXXXX..XX.] duration: 3.5204209s trace: https://jaeger.dev.storj.io/trace/294cb91aafb2e85b FAIL (2025-12-15 08:20:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520455316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.......X..] duration: 24.116389967s trace: https://jaeger.dev.storj.io/trace/5d3d0371eac0fbf2 FAIL (2025-12-15 08:19:39 UTC) orbiter warning: context deadline exceeded Duration: 24.116463978s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)