Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X..X...........X..X....X..X........X.....X.......XX.....] duration: 22.329308909s trace: https://jaeger.dev.storj.io/trace/1c72b2586479df7b FAIL (2026-01-12 16:21:43 UTC) orbiter warning: context deadline exceeded Duration: 22.329336571s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X....X..............X.X...........X...........] duration: 17.23137241s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:21:40 UTC) orbiter warning: context deadline exceeded Duration: 17.231421011s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXXX] duration: 5.002298477s trace: https://jaeger.dev.storj.io/trace/7f5f369481dc6be0 FAIL (2026-01-12 16:21:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002327993s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X..X...........X..X....X..X........X.....X.......XX....] duration: 22.313554648s trace: https://jaeger.dev.storj.io/trace/1c72b2586479df7b FAIL (2026-01-12 16:21:01 UTC) orbiter warning: context deadline exceeded Duration: 22.313582832s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X..............X....X..............X.X...........X..........] duration: 18.29604431s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:20:50 UTC) orbiter warning: context deadline exceeded Duration: 18.29607068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.965945413s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 5.965973466s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......X..X...........X..X....X..X........X.....X.......XX...] duration: 17.831743488s trace: https://jaeger.dev.storj.io/trace/1c72b2586479df7b FAIL (2026-01-12 16:20:27 UTC) orbiter warning: context deadline exceeded Duration: 17.831771281s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXXXXXXXXX] duration: 5.000988007s trace: https://jaeger.dev.storj.io/trace/7ef4e22e42069b6c FAIL (2026-01-12 16:20:24 UTC) orbiter warning: context deadline exceeded Duration: 5.001011129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXXXXXXXX] duration: 5.001888552s trace: https://jaeger.dev.storj.io/trace/5da45e6d170431d0 FAIL (2026-01-12 16:19:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001910375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XX..............X....X..............X.X...........X.........] duration: 16.666028437s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:19:56 UTC) orbiter warning: context deadline exceeded Duration: 16.666042753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.82486032s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:19:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 5.824949791s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X..X...........X..X....X..X........X.....X.......XX..] duration: 26.757445991s trace: https://jaeger.dev.storj.io/trace/1c72b2586479df7b FAIL (2026-01-12 16:19:31 UTC) orbiter warning: context deadline exceeded Duration: 26.757457893s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXXXXXXX] duration: 5.001669269s trace: https://jaeger.dev.storj.io/trace/2ee32b9564fb1bdc FAIL (2026-01-12 16:19:24 UTC) orbiter warning: context deadline exceeded Duration: 5.001691149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.XX..............X....X..............X.X...........X........] duration: 24.652909067s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:19:00 UTC) orbiter warning: context deadline exceeded Duration: 24.652933292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.830464087s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:18:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 5.830491078s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXXXXXX] duration: 5.000775927s trace: https://jaeger.dev.storj.io/trace/548937cc497375a7 FAIL (2026-01-12 16:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000799793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.922439233s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:18:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 5.922460834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..XX..............X....X..............X.X...........X.......] duration: 8.838462991s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:18:20 UTC) orbiter warning: context deadline exceeded Duration: 8.83848961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.......X..X...........X..X....X..X........X.....X.......XX.] duration: 17.47113007s trace: https://jaeger.dev.storj.io/trace/1c72b2586479df7b FAIL (2026-01-12 16:18:10 UTC) orbiter warning: context deadline exceeded Duration: 17.471179033s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXXXXX] duration: 5.001205575s trace: https://jaeger.dev.storj.io/trace/60fbb0c67a73436c FAIL (2026-01-12 16:17:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001228909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.96335565s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:17:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 5.963383914s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.......X..X...........X..X....X..X........X.....X.......XX] duration: 30.001466878s trace: https://jaeger.dev.storj.io/trace/1c72b2586479df7b FAIL (2026-01-12 16:17:24 UTC) orbiter warning: context deadline exceeded Duration: 30.001489462s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...XX..............X....X..............X.X...........X......] duration: 8.503015993s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:17:09 UTC) orbiter warning: context deadline exceeded Duration: 8.503041992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.89278544s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:17:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 5.892811491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXXXX] duration: 5.000889381s trace: https://jaeger.dev.storj.io/trace/ad268908866b924 FAIL (2026-01-12 16:16:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000913586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.......X..X...........X..X....X..X........X.....X.......X] duration: 30.001444024s trace: https://jaeger.dev.storj.io/trace/5b00e329283dba9c FAIL (2026-01-12 16:16:51 UTC) orbiter warning: context deadline exceeded Duration: 30.001455828s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X...XX..............X....X..............X.X...........X.....] duration: 19.190760988s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:16:08 UTC) orbiter warning: context deadline exceeded Duration: 19.190809619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.006304618s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:16:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 6.006333002s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.......X..X...........X..X....X..X........X.....X.......] duration: 22.163812685s trace: https://jaeger.dev.storj.io/trace/78bd58a60b4e4d96 FAIL (2026-01-12 16:15:47 UTC) orbiter warning: context deadline exceeded Duration: 22.163842501s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXXX] duration: 5.00035735s trace: https://jaeger.dev.storj.io/trace/5d45f8451fd27e55 FAIL (2026-01-12 16:15:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00037935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.880945237s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 5.880970124s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X...X.......X..X...........X..X....X..X........X.....X......] duration: 20.75833771s trace: https://jaeger.dev.storj.io/trace/78bd58a60b4e4d96 FAIL (2026-01-12 16:15:03 UTC) orbiter warning: context deadline exceeded Duration: 20.758440636s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X...XX..............X....X..............X.X...........X....] duration: 17.735781676s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:15:01 UTC) orbiter warning: context deadline exceeded Duration: 17.735808317s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 18.312144129s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:14:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 18.312169338s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXXX] duration: 5.001080425s trace: https://jaeger.dev.storj.io/trace/12064aa127be79e FAIL (2026-01-12 16:14:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001121633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X...X.......X..X...........X..X....X..X........X.....X.....] duration: 19.482793224s trace: https://jaeger.dev.storj.io/trace/78bd58a60b4e4d96 FAIL (2026-01-12 16:14:38 UTC) orbiter warning: context deadline exceeded Duration: 19.482820797s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXXX] duration: 5.001203248s trace: https://jaeger.dev.storj.io/trace/4afd8e902c97fc00 FAIL (2026-01-12 16:14:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001223226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002600415s trace: https://jaeger.dev.storj.io/trace/599858a692610b96 FAIL (2026-01-12 16:14:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ) context deadline exceeded ======LOGS====== Jan 12 16:13:54.279464385 (912h44m54.15054743s) [upload=798874]: Starting upload Jan 12 16:13:54.279521123 (912h44m54.150604158s) [upload=798874]: Done starting upload Jan 12 16:13:54.279599412 (912h44m54.150682447s) [upload=798874]: Uploading segments... Jan 12 16:13:54.279696237 (912h44m54.150779282s) [upload=798874]: Got next segment. Inline: false Jan 12 16:13:54.279730412 (912h44m54.150813447s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 16:13:54.279734960 (912h44m54.150817995s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 16:13:54.548547286 (912h44m54.419630331s) [seg_pos={0 0}, upload=798874]: Begin upload segment... Jan 12 16:13:54.548606880 (912h44m54.419689925s) [seg_pos={0 0}, upload=798874]: Done begin upload segment. Jan 12 16:13:54.548638901 (912h44m54.419721946s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.548799647 (912h44m54.419882692s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.549886287 (912h44m54.420969332s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550132225 (912h44m54.42121527s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550292119 (912h44m54.421375154s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550660450 (912h44m54.421743496s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.550716416 (912h44m54.421799461s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551116509 (912h44m54.422199544s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551184108 (912h44m54.422267153s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551531910 (912h44m54.422614945s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.551740587 (912h44m54.422823632s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552008777 (912h44m54.423091822s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552265386 (912h44m54.423348421s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552429188 (912h44m54.423512233s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552781659 (912h44m54.423864704s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:54.552905024 (912h44m54.423988069s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Uploading piece... Jan 12 16:13:56.385567454 (912h44m56.256650489s) [upload=798874]: Next returned nil segment Jan 12 16:13:56.385582582 (912h44m56.256665617s) [upload=798874]: Waiting for error group managing segments... Jan 12 16:13:56.685979738 (912h44m56.557062773s) [noise=true , address=145.79.227.4:4187 , piece_id=QUBVYLZ3RKPUD5XA2K3LMRNK4HRDH3HJFJZKUFRK7T6PQGZUR42Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.690328889 (912h44m56.561411924s) [noise=true , address=145.79.227.46:4603 , piece_id=CO5QQBCKJ6ACBI36AHGPPDTSTHWNCSTFWRPDBDARS53AA35I6UAA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.742055600 (912h44m56.613138635s) [noise=true , address=77.90.31.21:4877 , piece_id=5RUYQTLNIQJPOF7XPHUHJOS5Z4LBP363DEONTAF32QJL5AY7GRWQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.783984245 (912h44m56.65506728s) [noise=true , address=145.79.227.38:4977 , piece_id=XDMUID2SVUZFQEKHALZTP7AG7AKZBQ6ZW5NOQOQYAQBVJDSM6PUA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.809240525 (912h44m56.68032356s) [noise=true , address=77.90.31.12:4590 , piece_id=PHCRAMVR6EY572DLVKRCVAMMBMKELZGVDIAGTY6DJROYBXBQFSYA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.828621668 (912h44m56.699704703s) [noise=true , address=45.91.116.7:4009 , piece_id=MUAHJG6DCYJRTCL2PXXKGQJK2DOPD4475QOQ7TJ3JQQSE2GXY7BA, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840337295 (912h44m56.71142033s) [noise=true , address=45.91.116.8:4047 , piece_id=64W3SSIVKZGGW4JWOV5YDJEQ3YXWZAX4BY7OLBRBLL7IZ46HOOCQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.840422847 (912h44m56.711505882s) [noise=true , address=77.90.31.11:4555 , piece_id=BUCGWRZF7TND2QGF7PMTZTWUEASTNDYFSABS5YZGN3JY6GHAQ6CQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:56.893893409 (912h44m56.764976444s) [noise=true , address=45.91.116.6:4974 , piece_id=QTN2UHTOGMQ4UYEU6HC4MLDPCDHZ5NHYONEOHBAH525HRPFFJSTQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.197684167 (912h44m57.068767212s) [noise=true , address=145.79.227.15:4875 , piece_id=DXRGKPRU4LHINJI46PG5WER45R4BLLLOBTEDYTUDOCWKXJNL6A4Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.275329689 (912h44m57.146412724s) [noise=true , address=45.91.116.13:4153 , piece_id=WCABSIT7J3TDZTEQOPTQKJCSXB6MQQOPU4PBHQ6J52F3CQMVRMSQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.590395655 (912h44m57.46147869s) [noise=true , address=145.79.227.19:4078 , piece_id=QBOXXSW5LL26CIYW4QVUETA5WM3TA5HOX5A45DW6I6HBNSXPF7AQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598417059 (912h44m57.469500094s) [noise=true , address=145.79.227.50:4375 , piece_id=IPNONLZUCGN7HA3HGTLCRBZLSDGUVNCHI7RT27FQXPQDB2VID43Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: <nil> Jan 12 16:13:57.598436405 (912h44m57.46951944s) [seg_pos={0 0}, upload=798874]: Segment reached optimal threshold of 13 pieces. Jan 12 16:13:57.598597883 (912h44m57.469680918s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54942->77.90.31.10:4517: use of closed network connection Jan 12 16:13:57.598603193 (912h44m57.469686228s) [noise=true , address=77.90.31.10:4517 , piece_id=JXGKGQVXC534EX6KW24CVG5EVKLB4BKQHXFRC22Y5SPGJXK4G46Q, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598851105 (912h44m57.46993414s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598885079 (912h44m57.469968124s) [noise=true , address=77.90.31.19:4844 , piece_id=QIRIK7XDVJK5WLJQE5HMQQ6BXNDNFCRFKI3NRNY7AVYELGHL3WMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.598956405 (912h44m57.47003945s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 12 16:13:57.598964540 (912h44m57.470047575s) [noise=true , address=145.79.227.35:4481 , piece_id=Q7XWQEXQTFLE5JHW2NI7GGCVKLU4OJ7MCO7PVNJ5MSFSOIKE4JMQ, seg_pos={0 0}, upload=798874]: Long tail cancellation due to optimal threshold Jan 12 16:13:57.599103385 (912h44m57.47018642s) [seg_pos={0 0}, upload=798874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 16:13:57.599247689 (912h44m57.470330724s) [upload=798874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 16:13:57.599253842 (912h44m57.470336887s) [upload=798874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 16:13:58.241959194 (912h44m58.113042229s) [upload=798874]: Done uploading segments... Jan 12 16:13:58.241983461 (912h44m58.113066496s) [upload=798874]: Upload finished. err: <nil> Duration: 30.002620713s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X...XX..............X....X..............X.X...........X...] duration: 17.866635474s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:14:09 UTC) orbiter warning: context deadline exceeded Duration: 17.866660443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X...X.......X..X...........X..X....X..X........X.....X....] duration: 29.011294336s trace: https://jaeger.dev.storj.io/trace/78bd58a60b4e4d96 FAIL (2026-01-12 16:14:08 UTC) orbiter warning: context deadline exceeded Duration: 29.01131188s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXXX] duration: 5.0012772s trace: https://jaeger.dev.storj.io/trace/fba1ddea3d65545 FAIL (2026-01-12 16:13:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001299873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X...X.......X..X...........X..X....X..X........X.....X...] duration: 19.632289636s trace: https://jaeger.dev.storj.io/trace/78bd58a60b4e4d96 FAIL (2026-01-12 16:13:18 UTC) orbiter warning: context deadline exceeded Duration: 19.632319583s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXXX] duration: 5.001381334s trace: https://jaeger.dev.storj.io/trace/47ab8f5a0db238a2 FAIL (2026-01-12 16:12:58 UTC) orbiter warning: context deadline exceeded Duration: 5.001427583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X...XX..............X....X..............X.X...........X..] duration: 7.917124252s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:12:47 UTC) orbiter warning: context deadline exceeded Duration: 7.917163586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X...X.......X..X...........X..X....X..X........X.....X..] duration: 29.843782042s trace: https://jaeger.dev.storj.io/trace/78bd58a60b4e4d96 FAIL (2026-01-12 16:12:41 UTC) orbiter warning: context deadline exceeded Duration: 29.843821407s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXXX] duration: 5.000921932s trace: https://jaeger.dev.storj.io/trace/2fc38b8a4227fb2e FAIL (2026-01-12 16:12:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000943085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX.XXXXX.XXXXXXXXXXXXXXX..XXXX] duration: 5.001559448s trace: https://jaeger.dev.storj.io/trace/7107c0f841713935 FAIL (2026-01-12 16:12:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001585067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....X...XX..............X....X..............X.X...........X.] duration: 20.698805733s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:12:05 UTC) orbiter warning: context deadline exceeded Duration: 20.6988196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X...X.......X..X...........X..X....X..X........X.....X.] duration: 14.969026609s trace: https://jaeger.dev.storj.io/trace/78bd58a60b4e4d96 FAIL (2026-01-12 16:11:55 UTC) orbiter warning: context deadline exceeded Duration: 14.969053019s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X...XX..............X....X..............X.X...........X] duration: 1m0.00157506s trace: https://jaeger.dev.storj.io/trace/116a4062b0cb68f0 FAIL (2026-01-12 16:11:44 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001584468s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)