Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................X.............X........X............] duration: 2.389473079s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:30:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.389492075s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.............X......] duration: 675.868569ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:30:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 675.89506ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...X...................] duration: 770.63468ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:30:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 770.662492ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X.X.X...........X................] duration: 5.935366978s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:30:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.935391265s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.....XX....X.....X................X......X.X..X.X.........] duration: 16.385277871s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 16.385321815s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.............X.....] duration: 774.194456ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:29:49 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 774.223212ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...X..................] duration: 796.669302ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:29:48 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 796.691646ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.............X........X...........] duration: 2.402305898s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:29:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.402331566s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...X.................] duration: 723.656254ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:29:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 723.683407ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X.X.X...........X...............] duration: 6.366509758s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:29:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.366537913s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.............X........X..........] duration: 2.317528515s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:28:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.317554335s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.............X....] duration: 672.90194ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:28:52 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 672.944602ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X.....XX....X.....X................X......X.X..X.X........] duration: 13.473982343s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:28:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 13.47402261s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...X................] duration: 758.258533ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:28:29 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 758.285745ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.............X........X.........] duration: 2.290415316s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:28:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.290437289s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X.X.X...........X..............] duration: 5.745584513s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:28:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.745606264s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 704.456106ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:28:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 704.503365ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.............X...] duration: 704.826929ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:27:54 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 704.914355ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.X.....XX....X.....X................X......X.X..X.X.......] duration: 5.846771662s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:27:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 5.846797431s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X.X.X...........X.............] duration: 5.727489748s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:27:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.727518373s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X.............X........X........] duration: 2.283101582s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:27:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.28313192s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.............X........X.......] duration: 2.281782023s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:27:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.281883084s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.............X..] duration: 764.715178ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:27:05 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 764.739293ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 711.899829ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:27:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 711.926719ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X.X.X...........X............] duration: 12.919709017s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:26:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.91977399s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.X.....XX....X.....X................X......X.X..X.X......] duration: 5.82494835s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:26:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 5.824973179s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X.X.X...........X...........] duration: 5.967605073s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:26:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.967635161s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.............X.] duration: 789.500472ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:26:18 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 789.514368ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 748.600885ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:26:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 748.626332ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.............X] duration: 2m0.000217207s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/22f6d3f91ccd40e5 FAIL (2025-12-21 16:26:17 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.000243277s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.X.....XX....X.....X................X......X.X..X.X.....] duration: 5.773055196s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:26:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 5.773083911s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.............X........X......] duration: 2.214653366s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:26:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.214697859s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X.X.X...........X..........] duration: 6.011583926s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:25:53 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.011638509s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X...X.X.....XX....X.....X................X......X.X..X.X....] duration: 5.763611176s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:25:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 5.763636304s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.............X........X.....] duration: 2.305137523s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:25:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.305185445s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 769.78956ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:25:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 769.818204ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 732.875229ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:25:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 732.901308ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [................................X.............X........X....] duration: 2.454501765s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:25:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.454541972s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX...X.X.....XX....X.....X................X......X.X..X.X...] duration: 5.818283785s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:25:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 5.818309043s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X.X.X...........X.........] duration: 5.938935101s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:24:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.938964146s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 734.972473ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:24:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 735.001618ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX...X.X.....XX....X.....X................X......X.X..X.X..] duration: 5.952607474s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:24:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 5.952632121s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.............X........X...] duration: 2.185861089s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:24:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.185891178s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X.X.X...........X........] duration: 5.82352501s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:23:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.823543845s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 754.613582ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:23:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 754.637356ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 726.081373ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/30426dd147165f04 FAIL (2025-12-21 16:23:43 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 726.110058ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 736.569895ms trace: https://jaeger.dev.storj.io/trace/2a2773beab466b5e FAIL (2025-12-21 16:23:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 736.601145ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX...X.X.....XX....X.....X................X......X.X..X.X.] duration: 5.923133675s trace: https://jaeger.dev.storj.io/trace/19246c87722525fb FAIL (2025-12-21 16:23:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ) context deadline exceeded; piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ) context deadline exceeded; piecestore: (Node ID: 12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, Piece ID: KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ) context deadline exceeded ======LOGS====== Dec 21 16:22:22.623896294 (384h53m22.494979339s) [upload=337814]: Starting upload Dec 21 16:22:22.624007646 (384h53m22.495090691s) [upload=337814]: Done starting upload Dec 21 16:22:22.624057460 (384h53m22.495140495s) [upload=337814]: Uploading segments... Dec 21 16:22:22.624252272 (384h53m22.495335317s) [upload=337814]: Got next segment. Inline: false Dec 21 16:22:22.624290274 (384h53m22.495373309s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 16:22:22.624295624 (384h53m22.495378669s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 16:22:23.149491229 (384h53m23.020574274s) [seg_pos={0 0}, upload=337814]: Begin upload segment... Dec 21 16:22:23.149545532 (384h53m23.020628577s) [seg_pos={0 0}, upload=337814]: Done begin upload segment. Dec 21 16:22:23.149560601 (384h53m23.020643646s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.149620885 (384h53m23.02070392s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.150637702 (384h53m23.021720737s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151098289 (384h53m23.022181324s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151442655 (384h53m23.0225257s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.151749800 (384h53m23.022832845s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152044030 (384h53m23.023127075s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152319805 (384h53m23.02340284s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.152730908 (384h53m23.023813953s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.153582208 (384h53m23.024665253s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154059287 (384h53m23.025142322s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.154875010 (384h53m23.025958055s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.155427973 (384h53m23.026511008s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156068634 (384h53m23.027151669s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.156739571 (384h53m23.027822606s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:23.157279859 (384h53m23.028362904s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Uploading piece... Dec 21 16:22:25.028882043 (384h53m24.899965078s) [upload=337814]: Next returned nil segment Dec 21 16:22:25.028895599 (384h53m24.899978634s) [upload=337814]: Waiting for error group managing segments... Dec 21 16:22:25.334609065 (384h53m25.2056921s) [noise=true , address=145.79.227.32:4834 , piece_id=ZGNXSG4MPBSFLC3NE5HTKKQKF5EINXJWZGB3CQ35AI73KZQ5AJHQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.336440582 (384h53m25.207523617s) [noise=true , address=145.79.227.35:4482 , piece_id=AQEN3CM34NDHUR6ZX7RXDWABIL5QBREUK44V6WBT2BTH4ZZLYKBQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.362945542 (384h53m25.234028577s) [noise=true , address=145.79.227.49:4339 , piece_id=VX3FDN7D75AHJ4KLVIA4CQXIESGPFU7JPHJKOW7VXOUPFNVQSKIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.386938348 (384h53m25.258021383s) [noise=true , address=145.79.227.11:4732 , piece_id=7CUURC4D5DIGJFVU2WV6NEQCZ3ZZVQIQKXN75KIY2NUGQJIO7B3A, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.429568022 (384h53m25.300651067s) [noise=true , address=77.90.31.15:4700 , piece_id=XUYFHSN7KMHYSO5Z4FUMGLBKT26ZTEYACMBHEVT64MY2ALXTRGMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.442526006 (384h53m25.313609051s) [noise=true , address=45.91.116.9:4264 , piece_id=CRDWMI4BTVMEJPP2V4ZQJPJIEVDHU7EK3AJLIRZSVKWOGPKBGRLQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.448390352 (384h53m25.319473387s) [noise=true , address=77.90.31.21:4880 , piece_id=2VUTSI253ZMOAGYQGI7LDQIXIN2FL6UWPZNBKFULWOPENDC3VXQQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.514932539 (384h53m25.386015584s) [noise=true , address=45.91.116.14:4192 , piece_id=K3MZTNA2MIMDHY6KSI2LXLFT4NBGDFL2H6P42ZSZJTEDIGHSEAWA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.538975762 (384h53m25.410058797s) [noise=true , address=77.90.31.13:4629 , piece_id=FK7HXOXHUDNWKZFV3ACCV5GLOXJNKETNSOL3M55TY722CMXPMNWQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:25.740602210 (384h53m25.611685245s) [noise=true , address=45.91.116.10:4081 , piece_id=X25NGUND6GLIOOPFXHDBPJYUJSWEKHK4VT75QZBKAJKO6BPRPPCA, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.207763298 (384h53m26.078846343s) [noise=true , address=45.91.116.4:4939 , piece_id=KM2OIE3BCXX2FOHTXGKY7IYGMVOZ5R55OZRSKUCGTJAGTTW7FORQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:26.432615873 (384h53m26.303698918s) [noise=true , address=145.79.227.7:4295 , piece_id=CGGO6C2MVZHZZW4SH3TIW3BBHGXMCNILLQHBEIBWEWRNE7DFW5NQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:27.119282132 (384h53m26.990365177s) [noise=true , address=77.90.31.16:4735 , piece_id=JXMB2VJFQR5GTVRMXB3IPB3YQSEM7WOYZZCH6VL4YJMVV2H7GE4Q, seg_pos={0 0}, upload=337814]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 21 16:22:28.321776799 (384h53m28.192859844s) [noise=true , address=145.79.227.42:4121 , piece_id=5SDLAQNLM3VMEONT7W7DS72XL3DRM2RJQMEOH5X2VUHDBLZOFWFQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: <nil> Dec 21 16:22:28.321837905 (384h53m28.19292095s) [seg_pos={0 0}, upload=337814]: Segment reached optimal threshold of 13 pieces. Dec 21 16:22:28.322092259 (384h53m28.193175294s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 21 16:22:28.322104042 (384h53m28.193187077s) [noise=true , address=145.79.227.27:4690 , piece_id=5BU2YL4D2TWKXRMOYWE65FITNWBWA534JQM6M6AYWTWFGGKJRLMQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322285488 (384h53m28.193368523s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: read tcp 188.166.225.170:44716->145.79.227.33:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 16:22:28.322294034 (384h53m28.193377069s) [noise=true , address=145.79.227.33:4869 , piece_id=QYYEMQJ77FG2VFD64UPAESHL6R5XH6QYLD3QMHDVAGEC6HSPJYIQ, seg_pos={0 0}, upload=337814]: Long tail cancellation due to optimal threshold Dec 21 16:22:28.322358517 (384h53m28.193441552s) [seg_pos={0 0}, upload=337814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 16:22:28.322470720 (384h53m28.193553765s) [upload=337814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 16:22:28.322478175 (384h53m28.19356121s) [upload=337814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 16:22:28.662727803 (384h53m28.533810838s) [upload=337814]: Done uploading segments... Dec 21 16:22:28.662748062 (384h53m28.533831107s) [upload=337814]: Upload finished. err: <nil> Duration: 5.923163893s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.............X........X..] duration: 2.245334518s trace: https://jaeger.dev.storj.io/trace/38b8715a7148a1d9 FAIL (2025-12-21 16:23:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.245367221s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X.X.X...........X.......] duration: 6.031845519s trace: https://jaeger.dev.storj.io/trace/59609e189a00d0ea FAIL (2025-12-21 16:22:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.031870466s edge gateway us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)