Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.X.........X.X..........X..X..X.XX.............X............] duration:
6.086808109s trace:
https://jaeger.dev.storj.io/trace/1f3c53a17b6f7fb6
FAIL (2026-01-07 15:15:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA) context deadline exceeded
======LOGS======
Jan 7 15:07:19.312082232 (791h38m19.183165267s) [upload=692421]: Starting upload
Jan 7 15:07:19.312207981 (791h38m19.183291046s) [upload=692421]: Done starting upload
Jan 7 15:07:19.312284578 (791h38m19.183367643s) [upload=692421]: Uploading segments...
Jan 7 15:07:19.312502482 (791h38m19.183585527s) [upload=692421]: Got next segment. Inline: false
Jan 7 15:07:19.312544662 (791h38m19.183627717s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:07:19.312550694 (791h38m19.183633739s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:07:19.569325994 (791h38m19.440409039s) [seg_pos={0 0}, upload=692421]: Begin upload segment...
Jan 7 15:07:19.569382582 (791h38m19.440465627s) [seg_pos={0 0}, upload=692421]: Done begin upload segment.
Jan 7 15:07:19.569492361 (791h38m19.440575406s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570231268 (791h38m19.441314313s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570501242 (791h38m19.441584277s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570740167 (791h38m19.441823202s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571014419 (791h38m19.442097464s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571272691 (791h38m19.442355736s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571546643 (791h38m19.442629688s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571846985 (791h38m19.44293002s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572147166 (791h38m19.443230201s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572390539 (791h38m19.443473584s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572632821 (791h38m19.443715856s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572895762 (791h38m19.443978797s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573185794 (791h38m19.444268829s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573558714 (791h38m19.444641749s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583179604 (791h38m19.454262639s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583602939 (791h38m19.454685975s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:21.192726317 (791h38m21.063809362s) [upload=692421]: Next returned nil segment
Jan 7 15:07:21.192743761 (791h38m21.063826806s) [upload=692421]: Waiting for error group managing segments...
Jan 7 15:07:21.573623815 (791h38m21.44470686s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.574742956 (791h38m21.445825991s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.576831663 (791h38m21.447914708s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.577347465 (791h38m21.448430511s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.594180047 (791h38m21.465263092s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.596898944 (791h38m21.467981989s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.597996954 (791h38m21.46907999s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.598323697 (791h38m21.469406752s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.619734787 (791h38m21.490817822s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.655444514 (791h38m21.526527559s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.759546089 (791h38m21.630629124s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.032297580 (791h38m21.903380625s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694966023 (791h38m22.566049068s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694991682 (791h38m22.566074717s) [seg_pos={0 0}, upload=692421]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:07:22.695216660 (791h38m22.566299705s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695226038 (791h38m22.566309083s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.695609408 (791h38m22.566692463s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695618646 (791h38m22.566701681s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696042873 (791h38m22.567125918s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43498->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:07:22.696052972 (791h38m22.567136027s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696076738 (791h38m22.567159783s) [seg_pos={0 0}, upload=692421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:07:22.696162061 (791h38m22.567245106s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:22.696168693 (791h38m22.567251738s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:23.002248074 (791h38m22.87333112s) [upload=692421]: Done uploading segments...
Jan 7 15:07:23.002274104 (791h38m22.873357139s) [upload=692421]: Upload finished. err: <nil>
Duration: 6.086834938s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
1.761797714s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:15:43 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 1.761817231s
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [...X........X.............X......XXX..X..X...X......X.......] duration:
13.847172273s trace:
https://jaeger.dev.storj.io/trace/23514c16b68dbbaa
FAIL (2026-01-07 15:15:36 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 13.847203373s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [XXXX.XX....XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXX] duration:
30.000976601s trace:
https://jaeger.dev.storj.io/trace/18d097c8c6e93323
FAIL (2026-01-07 15:15:28 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000988524s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..X.........X.X..........X..X..X.XX.............X...........] duration:
5.848528558s trace:
https://jaeger.dev.storj.io/trace/1f3c53a17b6f7fb6
FAIL (2026-01-07 15:15:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA) context deadline exceeded
======LOGS======
Jan 7 15:07:19.312082232 (791h38m19.183165267s) [upload=692421]: Starting upload
Jan 7 15:07:19.312207981 (791h38m19.183291046s) [upload=692421]: Done starting upload
Jan 7 15:07:19.312284578 (791h38m19.183367643s) [upload=692421]: Uploading segments...
Jan 7 15:07:19.312502482 (791h38m19.183585527s) [upload=692421]: Got next segment. Inline: false
Jan 7 15:07:19.312544662 (791h38m19.183627717s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:07:19.312550694 (791h38m19.183633739s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:07:19.569325994 (791h38m19.440409039s) [seg_pos={0 0}, upload=692421]: Begin upload segment...
Jan 7 15:07:19.569382582 (791h38m19.440465627s) [seg_pos={0 0}, upload=692421]: Done begin upload segment.
Jan 7 15:07:19.569492361 (791h38m19.440575406s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570231268 (791h38m19.441314313s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570501242 (791h38m19.441584277s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570740167 (791h38m19.441823202s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571014419 (791h38m19.442097464s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571272691 (791h38m19.442355736s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571546643 (791h38m19.442629688s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571846985 (791h38m19.44293002s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572147166 (791h38m19.443230201s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572390539 (791h38m19.443473584s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572632821 (791h38m19.443715856s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572895762 (791h38m19.443978797s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573185794 (791h38m19.444268829s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573558714 (791h38m19.444641749s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583179604 (791h38m19.454262639s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583602939 (791h38m19.454685975s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:21.192726317 (791h38m21.063809362s) [upload=692421]: Next returned nil segment
Jan 7 15:07:21.192743761 (791h38m21.063826806s) [upload=692421]: Waiting for error group managing segments...
Jan 7 15:07:21.573623815 (791h38m21.44470686s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.574742956 (791h38m21.445825991s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.576831663 (791h38m21.447914708s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.577347465 (791h38m21.448430511s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.594180047 (791h38m21.465263092s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.596898944 (791h38m21.467981989s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.597996954 (791h38m21.46907999s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.598323697 (791h38m21.469406752s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.619734787 (791h38m21.490817822s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.655444514 (791h38m21.526527559s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.759546089 (791h38m21.630629124s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.032297580 (791h38m21.903380625s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694966023 (791h38m22.566049068s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694991682 (791h38m22.566074717s) [seg_pos={0 0}, upload=692421]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:07:22.695216660 (791h38m22.566299705s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695226038 (791h38m22.566309083s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.695609408 (791h38m22.566692463s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695618646 (791h38m22.566701681s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696042873 (791h38m22.567125918s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43498->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:07:22.696052972 (791h38m22.567136027s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696076738 (791h38m22.567159783s) [seg_pos={0 0}, upload=692421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:07:22.696162061 (791h38m22.567245106s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:22.696168693 (791h38m22.567251738s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:23.002248074 (791h38m22.87333112s) [upload=692421]: Done uploading segments...
Jan 7 15:07:23.002274104 (791h38m22.873357139s) [upload=692421]: Upload finished. err: <nil>
Duration: 5.848555178s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
8.273941354s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:15:08 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 8.273969067s
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.002256215s trace:
https://jaeger.dev.storj.io/trace/1d02c123b978d2b0
FAIL (2026-01-07 15:15:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002295869s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
53 out of 60 is failed (threshold=3) [XXXXX.XX....XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXX] duration:
30.000891734s trace:
https://jaeger.dev.storj.io/trace/349a38fd75cfba5b
FAIL (2026-01-07 15:14:58 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000913916s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...X.........X.X..........X..X..X.XX.............X..........] duration:
5.792368041s trace:
https://jaeger.dev.storj.io/trace/1f3c53a17b6f7fb6
FAIL (2026-01-07 15:14:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA) context deadline exceeded
======LOGS======
Jan 7 15:07:19.312082232 (791h38m19.183165267s) [upload=692421]: Starting upload
Jan 7 15:07:19.312207981 (791h38m19.183291046s) [upload=692421]: Done starting upload
Jan 7 15:07:19.312284578 (791h38m19.183367643s) [upload=692421]: Uploading segments...
Jan 7 15:07:19.312502482 (791h38m19.183585527s) [upload=692421]: Got next segment. Inline: false
Jan 7 15:07:19.312544662 (791h38m19.183627717s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:07:19.312550694 (791h38m19.183633739s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:07:19.569325994 (791h38m19.440409039s) [seg_pos={0 0}, upload=692421]: Begin upload segment...
Jan 7 15:07:19.569382582 (791h38m19.440465627s) [seg_pos={0 0}, upload=692421]: Done begin upload segment.
Jan 7 15:07:19.569492361 (791h38m19.440575406s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570231268 (791h38m19.441314313s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570501242 (791h38m19.441584277s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570740167 (791h38m19.441823202s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571014419 (791h38m19.442097464s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571272691 (791h38m19.442355736s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571546643 (791h38m19.442629688s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571846985 (791h38m19.44293002s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572147166 (791h38m19.443230201s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572390539 (791h38m19.443473584s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572632821 (791h38m19.443715856s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572895762 (791h38m19.443978797s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573185794 (791h38m19.444268829s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573558714 (791h38m19.444641749s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583179604 (791h38m19.454262639s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583602939 (791h38m19.454685975s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:21.192726317 (791h38m21.063809362s) [upload=692421]: Next returned nil segment
Jan 7 15:07:21.192743761 (791h38m21.063826806s) [upload=692421]: Waiting for error group managing segments...
Jan 7 15:07:21.573623815 (791h38m21.44470686s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.574742956 (791h38m21.445825991s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.576831663 (791h38m21.447914708s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.577347465 (791h38m21.448430511s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.594180047 (791h38m21.465263092s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.596898944 (791h38m21.467981989s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.597996954 (791h38m21.46907999s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.598323697 (791h38m21.469406752s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.619734787 (791h38m21.490817822s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.655444514 (791h38m21.526527559s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.759546089 (791h38m21.630629124s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.032297580 (791h38m21.903380625s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694966023 (791h38m22.566049068s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694991682 (791h38m22.566074717s) [seg_pos={0 0}, upload=692421]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:07:22.695216660 (791h38m22.566299705s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695226038 (791h38m22.566309083s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.695609408 (791h38m22.566692463s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695618646 (791h38m22.566701681s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696042873 (791h38m22.567125918s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43498->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:07:22.696052972 (791h38m22.567136027s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696076738 (791h38m22.567159783s) [seg_pos={0 0}, upload=692421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:07:22.696162061 (791h38m22.567245106s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:22.696168693 (791h38m22.567251738s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:23.002248074 (791h38m22.87333112s) [upload=692421]: Done uploading segments...
Jan 7 15:07:23.002274104 (791h38m22.873357139s) [upload=692421]: Upload finished. err: <nil>
Duration: 5.792391987s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [....X........X.............X......XXX..X..X...X......X......] duration:
15.729472658s trace:
https://jaeger.dev.storj.io/trace/23514c16b68dbbaa
FAIL (2026-01-07 15:14:36 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 15.72949969s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
16 out of 60 is failed (threshold=3) [...XX..X..........XX..X....X.X...XX..X.X..XX........X.....X.] duration:
15.60443183s trace:
https://jaeger.dev.storj.io/trace/56ef56e00fd0f2b6
FAIL (2026-01-07 15:14:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, Piece ID: BMF4YTYQMRELRQKNBM7IGP6RCR5T27H34KFCFBHYSIOGQ3IUI7WA) context deadline exceeded; piecestore: (Node ID: 12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, Piece ID: 4T6LJDZSPYJOSW5SSK2JVMCL67KQIOCZQX6RGOQB3YPEGLV6Z3MQ) context deadline exceeded; piecestore: (Node ID: 1vSQgozKbq7XjSYP5dUJjkrnTkWX4EFgBkruqUmdV3ehnoPfkr, Piece ID: MQCIXUQQGF22MKAWEXF3B2J7LDVD47SSDQAY6SHTUIRV6ACVI5JA) context deadline exceeded
======LOGS======
HRA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.624656623 (791h44m14.495739669s) [noise=true , address=145.79.227.34:4924 , piece_id=AQLL5H2YRUIIYCG7VOQS323FB5IG4ZLQ5YP4IVUNIRYZGANRHPNQ, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.625315999 (791h44m14.496399044s) [noise=true , address=145.79.227.40:4055 , piece_id=5HYKFSU3LQ4PG2Z5XPY6JO6PYBMUJILQZRORFDBGEA2J4AQZ7Z4A, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.625758271 (791h44m14.496841316s) [noise=true , address=145.79.227.44:4228 , piece_id=FYOAHNG22W5EHDD3GJQX5AYCELZ3G74BGOGOXA7C6LL4ZXDFUYCA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.626797059 (791h44m14.497880105s) [noise=true , address=45.91.116.12:4234 , piece_id=W4U7ZELVTVXDNUN5UY76JK6MXQGNWGVNI25EWTSZXBUQLWBUABKA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.628253383 (791h44m14.499336418s) [noise=true , address=77.90.31.13:4649 , piece_id=YPOI6HGNMA5FC52LU2TX36PK4DZOPX6IF4ENEMNZSK7I3746LNPA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.631116414 (791h44m14.502199449s) [noise=true , address=77.90.31.15:4723 , piece_id=DBTA7VVBVJD6T7YJXGEXBJ3PVOXFQX322SAGEBR5GZ5GLGBESBCQ, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.632490210 (791h44m14.503573245s) [noise=true , address=45.91.116.6:4992 , piece_id=Y2RFJLOJ47PGSTHBR45PMDQW35RNUMOU6RDPFVHJ7CQBBDZWPDSA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.633911065 (791h44m14.50499411s) [noise=true , address=64.49.13.169:4716 , piece_id=QEGIAXXLBCKJ3JT52UILF63ZRTXLMXQUI46IFQCWW7LZXJC74VBA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.635111881 (791h44m14.506194917s) [noise=true , address=64.49.13.155:4923 , piece_id=3HKJFKLIXSABO6LIXJKJE6AZITEDL5RTM2NO4G5QRNF4HGQWCH6Q, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.636346353 (791h44m14.507429398s) [noise=true , address=193.149.164.143:4700 , piece_id=BMF4YTYQMRELRQKNBM7IGP6RCR5T27H34KFCFBHYSIOGQ3IUI7WA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.638320883 (791h44m14.509403918s) [noise=true , address=45.91.116.13:4166 , piece_id=5X3CRAHSHOITUVEFCMK7PGAQPZKMNIDO5UOGEQNUWLZDSJE5EF5A, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.638804563 (791h44m14.509887608s) [noise=true , address=77.90.31.21:4897 , piece_id=4T6LJDZSPYJOSW5SSK2JVMCL67KQIOCZQX6RGOQB3YPEGLV6Z3MQ, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.640323165 (791h44m14.51140621s) [noise=true , address=145.79.227.30:4778 , piece_id=YNYSGJ3FOSFW6A7NNM7GBKW4J2CZSP4HC7CCF6AVCR75NSP5VXNA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.640522154 (791h44m14.511605189s) [noise=true , address=45.91.116.11:4124 , piece_id=MQCIXUQQGF22MKAWEXF3B2J7LDVD47SSDQAY6SHTUIRV6ACVI5JA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.642251516 (791h44m14.513334563s) [noise=true , address=45.91.116.4:4967 , piece_id=HFR753OBMNK6Z5J2PCDCQ3IP6FO2EIKUDK7PX7BB5MQU3NEG6H6A, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:16.074824126 (791h44m15.945907161s) [upload=692509]: Next returned nil segment
Jan 7 15:13:16.074835768 (791h44m15.945918813s) [upload=692509]: Waiting for error group managing segments...
Jan 7 15:13:16.485684768 (791h44m16.356767813s) [noise=true , address=145.79.227.40:4055 , piece_id=5HYKFSU3LQ4PG2Z5XPY6JO6PYBMUJILQZRORFDBGEA2J4AQZ7Z4A, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.503042269 (791h44m16.374125314s) [noise=true , address=145.79.227.44:4228 , piece_id=FYOAHNG22W5EHDD3GJQX5AYCELZ3G74BGOGOXA7C6LL4ZXDFUYCA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.506383822 (791h44m16.377466857s) [noise=true , address=64.49.13.169:4716 , piece_id=QEGIAXXLBCKJ3JT52UILF63ZRTXLMXQUI46IFQCWW7LZXJC74VBA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.524668990 (791h44m16.395752025s) [noise=true , address=145.79.227.30:4778 , piece_id=YNYSGJ3FOSFW6A7NNM7GBKW4J2CZSP4HC7CCF6AVCR75NSP5VXNA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.529232339 (791h44m16.400315384s) [noise=true , address=145.79.227.34:4924 , piece_id=AQLL5H2YRUIIYCG7VOQS323FB5IG4ZLQ5YP4IVUNIRYZGANRHPNQ, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.833058611 (791h44m16.704141646s) [noise=true , address=77.90.31.21:4897 , piece_id=4T6LJDZSPYJOSW5SSK2JVMCL67KQIOCZQX6RGOQB3YPEGLV6Z3MQ, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.934971178 (791h44m16.806054223s) [noise=true , address=45.91.116.4:4967 , piece_id=HFR753OBMNK6Z5J2PCDCQ3IP6FO2EIKUDK7PX7BB5MQU3NEG6H6A, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.005411588 (791h44m16.876494623s) [noise=true , address=45.91.116.12:4234 , piece_id=W4U7ZELVTVXDNUN5UY76JK6MXQGNWGVNI25EWTSZXBUQLWBUABKA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.246862997 (791h44m17.117946042s) [noise=true , address=45.91.116.11:4124 , piece_id=MQCIXUQQGF22MKAWEXF3B2J7LDVD47SSDQAY6SHTUIRV6ACVI5JA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.406305781 (791h44m17.277388826s) [noise=true , address=64.49.13.155:4914 , piece_id=YTZ5HWJ6SWC4AOU3ANCOBYI4IFTMZC7RW5YLRDOQBJE2WOK3KHRA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.422932852 (791h44m17.294015887s) [noise=true , address=145.79.227.32:4838 , piece_id=VMFXRPJRZVBTOXEXRHKCGMRPPNDIE2LGSLF3WLW4EDKAST76F2ZQ, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.433746783 (791h44m17.304829818s) [noise=true , address=64.49.13.155:4923 , piece_id=3HKJFKLIXSABO6LIXJKJE6AZITEDL5RTM2NO4G5QRNF4HGQWCH6Q, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.611056609 (791h44m17.482139654s) [noise=true , address=145.79.227.29:4944 , piece_id=YXP6IN4ZJ3LQHJVAZRHCGNJTLMUVPWZP7EAMQ3JNTPCUA46LDE6Q, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.766566744 (791h44m17.637649789s) [noise=true , address=193.149.164.143:4700 , piece_id=BMF4YTYQMRELRQKNBM7IGP6RCR5T27H34KFCFBHYSIOGQ3IUI7WA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.766588516 (791h44m17.637671561s) [seg_pos={0 0}, upload=692509]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:13:17.766765142 (791h44m17.637848187s) [noise=true , address=45.91.116.6:4992 , piece_id=Y2RFJLOJ47PGSTHBR45PMDQW35RNUMOU6RDPFVHJ7CQBBDZWPDSA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx): protocol: expected piece hash; context canceled; EOF
Jan 7 15:13:17.766772145 (791h44m17.63785519s) [noise=true , address=45.91.116.6:4992 , piece_id=Y2RFJLOJ47PGSTHBR45PMDQW35RNUMOU6RDPFVHJ7CQBBDZWPDSA, seg_pos={0 0}, upload=692509]: Long tail cancellation due to optimal threshold
Jan 7 15:13:17.767031969 (791h44m17.638115014s) [noise=true , address=77.90.31.13:4649 , piece_id=YPOI6HGNMA5FC52LU2TX36PK4DZOPX6IF4ENEMNZSK7I3746LNPA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF
Jan 7 15:13:17.767056707 (791h44m17.638139752s) [noise=true , address=77.90.31.13:4649 , piece_id=YPOI6HGNMA5FC52LU2TX36PK4DZOPX6IF4ENEMNZSK7I3746LNPA, seg_pos={0 0}, upload=692509]: Long tail cancellation due to optimal threshold
Jan 7 15:13:17.767122903 (791h44m17.638205938s) [noise=true , address=77.90.31.15:4723 , piece_id=DBTA7VVBVJD6T7YJXGEXBJ3PVOXFQX322SAGEBR5GZ5GLGBESBCQ, seg_pos={0 0}, upload=692509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: write tcp 188.166.225.170:38242->77.90.31.15:4723: use of closed network connection; write tcp 188.166.225.170:38242->77.90.31.15:4723: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38242->77.90.31.15:4723: use of closed network connection
Jan 7 15:13:17.767146117 (791h44m17.638229162s) [noise=true , address=77.90.31.15:4723 , piece_id=DBTA7VVBVJD6T7YJXGEXBJ3PVOXFQX322SAGEBR5GZ5GLGBESBCQ, seg_pos={0 0}, upload=692509]: Long tail cancellation due to optimal threshold
Jan 7 15:13:17.767429417 (791h44m17.638512452s) [noise=true , address=45.91.116.13:4166 , piece_id=5X3CRAHSHOITUVEFCMK7PGAQPZKMNIDO5UOGEQNUWLZDSJE5EF5A, seg_pos={0 0}, upload=692509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7): protocol: read tcp 188.166.225.170:40448->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:13:17.767436189 (791h44m17.638519244s) [noise=true , address=45.91.116.13:4166 , piece_id=5X3CRAHSHOITUVEFCMK7PGAQPZKMNIDO5UOGEQNUWLZDSJE5EF5A, seg_pos={0 0}, upload=692509]: Long tail cancellation due to optimal threshold
Jan 7 15:13:17.767455556 (791h44m17.638538591s) [seg_pos={0 0}, upload=692509]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:13:17.767558131 (791h44m17.638641176s) [upload=692509]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:13:17.767563641 (791h44m17.638646686s) [upload=692509]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:13:18.072299675 (791h44m17.94338271s) [upload=692509]: Done uploading segments...
Jan 7 15:13:18.072317630 (791h44m17.943400675s) [upload=692509]: Upload finished. err: <nil>
Duration: 15.604451218s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
16 out of 60 is failed (threshold=3) [....XX..X..........XX..X....X.X...XX..X.X..XX........X.....X] duration:
1m0.00259661s trace:
https://jaeger.dev.storj.io/trace/56ef56e00fd0f2b6
FAIL (2026-01-07 15:14:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, Piece ID: BMF4YTYQMRELRQKNBM7IGP6RCR5T27H34KFCFBHYSIOGQ3IUI7WA) context deadline exceeded; piecestore: (Node ID: 12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, Piece ID: 4T6LJDZSPYJOSW5SSK2JVMCL67KQIOCZQX6RGOQB3YPEGLV6Z3MQ) context deadline exceeded; piecestore: (Node ID: 1vSQgozKbq7XjSYP5dUJjkrnTkWX4EFgBkruqUmdV3ehnoPfkr, Piece ID: MQCIXUQQGF22MKAWEXF3B2J7LDVD47SSDQAY6SHTUIRV6ACVI5JA) context deadline exceeded
======LOGS======
HRA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.624656623 (791h44m14.495739669s) [noise=true , address=145.79.227.34:4924 , piece_id=AQLL5H2YRUIIYCG7VOQS323FB5IG4ZLQ5YP4IVUNIRYZGANRHPNQ, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.625315999 (791h44m14.496399044s) [noise=true , address=145.79.227.40:4055 , piece_id=5HYKFSU3LQ4PG2Z5XPY6JO6PYBMUJILQZRORFDBGEA2J4AQZ7Z4A, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.625758271 (791h44m14.496841316s) [noise=true , address=145.79.227.44:4228 , piece_id=FYOAHNG22W5EHDD3GJQX5AYCELZ3G74BGOGOXA7C6LL4ZXDFUYCA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.626797059 (791h44m14.497880105s) [noise=true , address=45.91.116.12:4234 , piece_id=W4U7ZELVTVXDNUN5UY76JK6MXQGNWGVNI25EWTSZXBUQLWBUABKA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.628253383 (791h44m14.499336418s) [noise=true , address=77.90.31.13:4649 , piece_id=YPOI6HGNMA5FC52LU2TX36PK4DZOPX6IF4ENEMNZSK7I3746LNPA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.631116414 (791h44m14.502199449s) [noise=true , address=77.90.31.15:4723 , piece_id=DBTA7VVBVJD6T7YJXGEXBJ3PVOXFQX322SAGEBR5GZ5GLGBESBCQ, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.632490210 (791h44m14.503573245s) [noise=true , address=45.91.116.6:4992 , piece_id=Y2RFJLOJ47PGSTHBR45PMDQW35RNUMOU6RDPFVHJ7CQBBDZWPDSA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.633911065 (791h44m14.50499411s) [noise=true , address=64.49.13.169:4716 , piece_id=QEGIAXXLBCKJ3JT52UILF63ZRTXLMXQUI46IFQCWW7LZXJC74VBA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.635111881 (791h44m14.506194917s) [noise=true , address=64.49.13.155:4923 , piece_id=3HKJFKLIXSABO6LIXJKJE6AZITEDL5RTM2NO4G5QRNF4HGQWCH6Q, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.636346353 (791h44m14.507429398s) [noise=true , address=193.149.164.143:4700 , piece_id=BMF4YTYQMRELRQKNBM7IGP6RCR5T27H34KFCFBHYSIOGQ3IUI7WA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.638320883 (791h44m14.509403918s) [noise=true , address=45.91.116.13:4166 , piece_id=5X3CRAHSHOITUVEFCMK7PGAQPZKMNIDO5UOGEQNUWLZDSJE5EF5A, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.638804563 (791h44m14.509887608s) [noise=true , address=77.90.31.21:4897 , piece_id=4T6LJDZSPYJOSW5SSK2JVMCL67KQIOCZQX6RGOQB3YPEGLV6Z3MQ, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.640323165 (791h44m14.51140621s) [noise=true , address=145.79.227.30:4778 , piece_id=YNYSGJ3FOSFW6A7NNM7GBKW4J2CZSP4HC7CCF6AVCR75NSP5VXNA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.640522154 (791h44m14.511605189s) [noise=true , address=45.91.116.11:4124 , piece_id=MQCIXUQQGF22MKAWEXF3B2J7LDVD47SSDQAY6SHTUIRV6ACVI5JA, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:14.642251516 (791h44m14.513334563s) [noise=true , address=45.91.116.4:4967 , piece_id=HFR753OBMNK6Z5J2PCDCQ3IP6FO2EIKUDK7PX7BB5MQU3NEG6H6A, seg_pos={0 0}, upload=692509]: Uploading piece...
Jan 7 15:13:16.074824126 (791h44m15.945907161s) [upload=692509]: Next returned nil segment
Jan 7 15:13:16.074835768 (791h44m15.945918813s) [upload=692509]: Waiting for error group managing segments...
Jan 7 15:13:16.485684768 (791h44m16.356767813s) [noise=true , address=145.79.227.40:4055 , piece_id=5HYKFSU3LQ4PG2Z5XPY6JO6PYBMUJILQZRORFDBGEA2J4AQZ7Z4A, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.503042269 (791h44m16.374125314s) [noise=true , address=145.79.227.44:4228 , piece_id=FYOAHNG22W5EHDD3GJQX5AYCELZ3G74BGOGOXA7C6LL4ZXDFUYCA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.506383822 (791h44m16.377466857s) [noise=true , address=64.49.13.169:4716 , piece_id=QEGIAXXLBCKJ3JT52UILF63ZRTXLMXQUI46IFQCWW7LZXJC74VBA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.524668990 (791h44m16.395752025s) [noise=true , address=145.79.227.30:4778 , piece_id=YNYSGJ3FOSFW6A7NNM7GBKW4J2CZSP4HC7CCF6AVCR75NSP5VXNA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.529232339 (791h44m16.400315384s) [noise=true , address=145.79.227.34:4924 , piece_id=AQLL5H2YRUIIYCG7VOQS323FB5IG4ZLQ5YP4IVUNIRYZGANRHPNQ, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.833058611 (791h44m16.704141646s) [noise=true , address=77.90.31.21:4897 , piece_id=4T6LJDZSPYJOSW5SSK2JVMCL67KQIOCZQX6RGOQB3YPEGLV6Z3MQ, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:16.934971178 (791h44m16.806054223s) [noise=true , address=45.91.116.4:4967 , piece_id=HFR753OBMNK6Z5J2PCDCQ3IP6FO2EIKUDK7PX7BB5MQU3NEG6H6A, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.005411588 (791h44m16.876494623s) [noise=true , address=45.91.116.12:4234 , piece_id=W4U7ZELVTVXDNUN5UY76JK6MXQGNWGVNI25EWTSZXBUQLWBUABKA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.246862997 (791h44m17.117946042s) [noise=true , address=45.91.116.11:4124 , piece_id=MQCIXUQQGF22MKAWEXF3B2J7LDVD47SSDQAY6SHTUIRV6ACVI5JA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.406305781 (791h44m17.277388826s) [noise=true , address=64.49.13.155:4914 , piece_id=YTZ5HWJ6SWC4AOU3ANCOBYI4IFTMZC7RW5YLRDOQBJE2WOK3KHRA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.422932852 (791h44m17.294015887s) [noise=true , address=145.79.227.32:4838 , piece_id=VMFXRPJRZVBTOXEXRHKCGMRPPNDIE2LGSLF3WLW4EDKAST76F2ZQ, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.433746783 (791h44m17.304829818s) [noise=true , address=64.49.13.155:4923 , piece_id=3HKJFKLIXSABO6LIXJKJE6AZITEDL5RTM2NO4G5QRNF4HGQWCH6Q, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.611056609 (791h44m17.482139654s) [noise=true , address=145.79.227.29:4944 , piece_id=YXP6IN4ZJ3LQHJVAZRHCGNJTLMUVPWZP7EAMQ3JNTPCUA46LDE6Q, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.766566744 (791h44m17.637649789s) [noise=true , address=193.149.164.143:4700 , piece_id=BMF4YTYQMRELRQKNBM7IGP6RCR5T27H34KFCFBHYSIOGQ3IUI7WA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: <nil>
Jan 7 15:13:17.766588516 (791h44m17.637671561s) [seg_pos={0 0}, upload=692509]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:13:17.766765142 (791h44m17.637848187s) [noise=true , address=45.91.116.6:4992 , piece_id=Y2RFJLOJ47PGSTHBR45PMDQW35RNUMOU6RDPFVHJ7CQBBDZWPDSA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ai9G3ULmwAy4zkKPUonML5fgxZGvJJgmqQeGfYHWqUKpehgtx): protocol: expected piece hash; context canceled; EOF
Jan 7 15:13:17.766772145 (791h44m17.63785519s) [noise=true , address=45.91.116.6:4992 , piece_id=Y2RFJLOJ47PGSTHBR45PMDQW35RNUMOU6RDPFVHJ7CQBBDZWPDSA, seg_pos={0 0}, upload=692509]: Long tail cancellation due to optimal threshold
Jan 7 15:13:17.767031969 (791h44m17.638115014s) [noise=true , address=77.90.31.13:4649 , piece_id=YPOI6HGNMA5FC52LU2TX36PK4DZOPX6IF4ENEMNZSK7I3746LNPA, seg_pos={0 0}, upload=692509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF
Jan 7 15:13:17.767056707 (791h44m17.638139752s) [noise=true , address=77.90.31.13:4649 , piece_id=YPOI6HGNMA5FC52LU2TX36PK4DZOPX6IF4ENEMNZSK7I3746LNPA, seg_pos={0 0}, upload=692509]: Long tail cancellation due to optimal threshold
Jan 7 15:13:17.767122903 (791h44m17.638205938s) [noise=true , address=77.90.31.15:4723 , piece_id=DBTA7VVBVJD6T7YJXGEXBJ3PVOXFQX322SAGEBR5GZ5GLGBESBCQ, seg_pos={0 0}, upload=692509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: write tcp 188.166.225.170:38242->77.90.31.15:4723: use of closed network connection; write tcp 188.166.225.170:38242->77.90.31.15:4723: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38242->77.90.31.15:4723: use of closed network connection
Jan 7 15:13:17.767146117 (791h44m17.638229162s) [noise=true , address=77.90.31.15:4723 , piece_id=DBTA7VVBVJD6T7YJXGEXBJ3PVOXFQX322SAGEBR5GZ5GLGBESBCQ, seg_pos={0 0}, upload=692509]: Long tail cancellation due to optimal threshold
Jan 7 15:13:17.767429417 (791h44m17.638512452s) [noise=true , address=45.91.116.13:4166 , piece_id=5X3CRAHSHOITUVEFCMK7PGAQPZKMNIDO5UOGEQNUWLZDSJE5EF5A, seg_pos={0 0}, upload=692509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7): protocol: read tcp 188.166.225.170:40448->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:13:17.767436189 (791h44m17.638519244s) [noise=true , address=45.91.116.13:4166 , piece_id=5X3CRAHSHOITUVEFCMK7PGAQPZKMNIDO5UOGEQNUWLZDSJE5EF5A, seg_pos={0 0}, upload=692509]: Long tail cancellation due to optimal threshold
Jan 7 15:13:17.767455556 (791h44m17.638538591s) [seg_pos={0 0}, upload=692509]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:13:17.767558131 (791h44m17.638641176s) [upload=692509]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:13:17.767563641 (791h44m17.638646686s) [upload=692509]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:13:18.072299675 (791h44m17.94338271s) [upload=692509]: Done uploading segments...
Jan 7 15:13:18.072317630 (791h44m17.943400675s) [upload=692509]: Upload finished. err: <nil>
Duration: 1m0.002619284s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
7.071218296s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:14:12 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 7.071264955s
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
34 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXX..XX.X.XXXX....X....XXX..X..XXX..X...XXX.X...] duration:
24.288509494s trace:
https://jaeger.dev.storj.io/trace/41ad92cdda135526
FAIL (2026-01-07 15:14:11 UTC)
orbiter warning: context deadline exceeded
Duration: 24.288533028s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [....X.........X.X..........X..X..X.XX.............X.........] duration:
6.218045183s trace:
https://jaeger.dev.storj.io/trace/1f3c53a17b6f7fb6
FAIL (2026-01-07 15:14:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA) context deadline exceeded
======LOGS======
Jan 7 15:07:19.312082232 (791h38m19.183165267s) [upload=692421]: Starting upload
Jan 7 15:07:19.312207981 (791h38m19.183291046s) [upload=692421]: Done starting upload
Jan 7 15:07:19.312284578 (791h38m19.183367643s) [upload=692421]: Uploading segments...
Jan 7 15:07:19.312502482 (791h38m19.183585527s) [upload=692421]: Got next segment. Inline: false
Jan 7 15:07:19.312544662 (791h38m19.183627717s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:07:19.312550694 (791h38m19.183633739s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:07:19.569325994 (791h38m19.440409039s) [seg_pos={0 0}, upload=692421]: Begin upload segment...
Jan 7 15:07:19.569382582 (791h38m19.440465627s) [seg_pos={0 0}, upload=692421]: Done begin upload segment.
Jan 7 15:07:19.569492361 (791h38m19.440575406s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570231268 (791h38m19.441314313s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570501242 (791h38m19.441584277s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570740167 (791h38m19.441823202s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571014419 (791h38m19.442097464s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571272691 (791h38m19.442355736s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571546643 (791h38m19.442629688s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571846985 (791h38m19.44293002s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572147166 (791h38m19.443230201s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572390539 (791h38m19.443473584s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572632821 (791h38m19.443715856s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572895762 (791h38m19.443978797s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573185794 (791h38m19.444268829s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573558714 (791h38m19.444641749s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583179604 (791h38m19.454262639s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583602939 (791h38m19.454685975s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:21.192726317 (791h38m21.063809362s) [upload=692421]: Next returned nil segment
Jan 7 15:07:21.192743761 (791h38m21.063826806s) [upload=692421]: Waiting for error group managing segments...
Jan 7 15:07:21.573623815 (791h38m21.44470686s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.574742956 (791h38m21.445825991s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.576831663 (791h38m21.447914708s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.577347465 (791h38m21.448430511s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.594180047 (791h38m21.465263092s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.596898944 (791h38m21.467981989s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.597996954 (791h38m21.46907999s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.598323697 (791h38m21.469406752s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.619734787 (791h38m21.490817822s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.655444514 (791h38m21.526527559s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.759546089 (791h38m21.630629124s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.032297580 (791h38m21.903380625s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694966023 (791h38m22.566049068s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694991682 (791h38m22.566074717s) [seg_pos={0 0}, upload=692421]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:07:22.695216660 (791h38m22.566299705s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695226038 (791h38m22.566309083s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.695609408 (791h38m22.566692463s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695618646 (791h38m22.566701681s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696042873 (791h38m22.567125918s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43498->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:07:22.696052972 (791h38m22.567136027s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696076738 (791h38m22.567159783s) [seg_pos={0 0}, upload=692421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:07:22.696162061 (791h38m22.567245106s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:22.696168693 (791h38m22.567251738s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:23.002248074 (791h38m22.87333112s) [upload=692421]: Done uploading segments...
Jan 7 15:07:23.002274104 (791h38m22.873357139s) [upload=692421]: Upload finished. err: <nil>
Duration: 6.218074509s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.002094276s trace:
https://jaeger.dev.storj.io/trace/12aabf13d543e909
FAIL (2026-01-07 15:14:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002119604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
52 out of 60 is failed (threshold=3) [.XXXXX.XX....XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXX] duration:
30.001472643s trace:
https://jaeger.dev.storj.io/trace/2ae12d160724386a
FAIL (2026-01-07 15:14:00 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001483814s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [X....X.........X.X..........X..X..X.XX.............X........] duration:
5.801461651s trace:
https://jaeger.dev.storj.io/trace/1f3c53a17b6f7fb6
FAIL (2026-01-07 15:13:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA) context deadline exceeded
======LOGS======
Jan 7 15:07:19.312082232 (791h38m19.183165267s) [upload=692421]: Starting upload
Jan 7 15:07:19.312207981 (791h38m19.183291046s) [upload=692421]: Done starting upload
Jan 7 15:07:19.312284578 (791h38m19.183367643s) [upload=692421]: Uploading segments...
Jan 7 15:07:19.312502482 (791h38m19.183585527s) [upload=692421]: Got next segment. Inline: false
Jan 7 15:07:19.312544662 (791h38m19.183627717s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:07:19.312550694 (791h38m19.183633739s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:07:19.569325994 (791h38m19.440409039s) [seg_pos={0 0}, upload=692421]: Begin upload segment...
Jan 7 15:07:19.569382582 (791h38m19.440465627s) [seg_pos={0 0}, upload=692421]: Done begin upload segment.
Jan 7 15:07:19.569492361 (791h38m19.440575406s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570231268 (791h38m19.441314313s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570501242 (791h38m19.441584277s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570740167 (791h38m19.441823202s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571014419 (791h38m19.442097464s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571272691 (791h38m19.442355736s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571546643 (791h38m19.442629688s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571846985 (791h38m19.44293002s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572147166 (791h38m19.443230201s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572390539 (791h38m19.443473584s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572632821 (791h38m19.443715856s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572895762 (791h38m19.443978797s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573185794 (791h38m19.444268829s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573558714 (791h38m19.444641749s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583179604 (791h38m19.454262639s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583602939 (791h38m19.454685975s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:21.192726317 (791h38m21.063809362s) [upload=692421]: Next returned nil segment
Jan 7 15:07:21.192743761 (791h38m21.063826806s) [upload=692421]: Waiting for error group managing segments...
Jan 7 15:07:21.573623815 (791h38m21.44470686s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.574742956 (791h38m21.445825991s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.576831663 (791h38m21.447914708s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.577347465 (791h38m21.448430511s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.594180047 (791h38m21.465263092s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.596898944 (791h38m21.467981989s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.597996954 (791h38m21.46907999s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.598323697 (791h38m21.469406752s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.619734787 (791h38m21.490817822s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.655444514 (791h38m21.526527559s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.759546089 (791h38m21.630629124s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.032297580 (791h38m21.903380625s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694966023 (791h38m22.566049068s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694991682 (791h38m22.566074717s) [seg_pos={0 0}, upload=692421]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:07:22.695216660 (791h38m22.566299705s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695226038 (791h38m22.566309083s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.695609408 (791h38m22.566692463s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695618646 (791h38m22.566701681s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696042873 (791h38m22.567125918s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43498->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:07:22.696052972 (791h38m22.567136027s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696076738 (791h38m22.567159783s) [seg_pos={0 0}, upload=692421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:07:22.696162061 (791h38m22.567245106s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:22.696168693 (791h38m22.567251738s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:23.002248074 (791h38m22.87333112s) [upload=692421]: Done uploading segments...
Jan 7 15:07:23.002274104 (791h38m22.873357139s) [upload=692421]: Upload finished. err: <nil>
Duration: 5.801480488s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00122052s trace:
https://jaeger.dev.storj.io/trace/23f37d842a59dd6
FAIL (2026-01-07 15:13:31 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001235319s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
52 out of 60 is failed (threshold=3) [X.XXXXX.XX....XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXX] duration:
30.001036594s trace:
https://jaeger.dev.storj.io/trace/5f8c57c9913b723e
FAIL (2026-01-07 15:13:30 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001047155s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [.....X........X.............X......XXX..X..X...X......X.....] duration:
10.407830086s trace:
https://jaeger.dev.storj.io/trace/23514c16b68dbbaa
FAIL (2026-01-07 15:13:30 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 10.407856458s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
35 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXX..XX.X.XXXX....X....XXX..X..XXX..X...XXX.X..] duration:
28.198444844s trace:
https://jaeger.dev.storj.io/trace/41ad92cdda135526
FAIL (2026-01-07 15:13:25 UTC)
orbiter warning: context deadline exceeded
Duration: 28.198569733s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.672717305s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:13:08 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 6.672747764s
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [.X.XXXXX.XX....XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XX] duration:
30.001643722s trace:
https://jaeger.dev.storj.io/trace/313579581f082cbd
FAIL (2026-01-07 15:13:00 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001660002s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
36 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXX..XX.X.XXXX....X....XXX..X..XXX..X...XXX.X.] duration:
34.966369828s trace:
https://jaeger.dev.storj.io/trace/41ad92cdda135526
FAIL (2026-01-07 15:12:57 UTC)
orbiter warning: context deadline exceeded
Duration: 34.966398282s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [......X........X.............X......XXX..X..X...X......X....] duration:
10.016010616s trace:
https://jaeger.dev.storj.io/trace/23514c16b68dbbaa
FAIL (2026-01-07 15:12:44 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 10.016036996s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [X.X.XXXXX.XX....XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.X] duration:
30.001222071s trace:
https://jaeger.dev.storj.io/trace/4dcea606129409c1
FAIL (2026-01-07 15:12:29 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001245748s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001497863s trace:
https://jaeger.dev.storj.io/trace/7b810415df91bac0
FAIL (2026-01-07 15:12:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001523812s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.X....X.........X.X..........X..X..X.XX.............X.......] duration:
6.119126187s trace:
https://jaeger.dev.storj.io/trace/1f3c53a17b6f7fb6
FAIL (2026-01-07 15:12:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA) context deadline exceeded
======LOGS======
Jan 7 15:07:19.312082232 (791h38m19.183165267s) [upload=692421]: Starting upload
Jan 7 15:07:19.312207981 (791h38m19.183291046s) [upload=692421]: Done starting upload
Jan 7 15:07:19.312284578 (791h38m19.183367643s) [upload=692421]: Uploading segments...
Jan 7 15:07:19.312502482 (791h38m19.183585527s) [upload=692421]: Got next segment. Inline: false
Jan 7 15:07:19.312544662 (791h38m19.183627717s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:07:19.312550694 (791h38m19.183633739s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:07:19.569325994 (791h38m19.440409039s) [seg_pos={0 0}, upload=692421]: Begin upload segment...
Jan 7 15:07:19.569382582 (791h38m19.440465627s) [seg_pos={0 0}, upload=692421]: Done begin upload segment.
Jan 7 15:07:19.569492361 (791h38m19.440575406s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570231268 (791h38m19.441314313s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570501242 (791h38m19.441584277s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570740167 (791h38m19.441823202s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571014419 (791h38m19.442097464s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571272691 (791h38m19.442355736s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571546643 (791h38m19.442629688s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571846985 (791h38m19.44293002s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572147166 (791h38m19.443230201s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572390539 (791h38m19.443473584s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572632821 (791h38m19.443715856s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572895762 (791h38m19.443978797s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573185794 (791h38m19.444268829s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573558714 (791h38m19.444641749s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583179604 (791h38m19.454262639s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583602939 (791h38m19.454685975s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:21.192726317 (791h38m21.063809362s) [upload=692421]: Next returned nil segment
Jan 7 15:07:21.192743761 (791h38m21.063826806s) [upload=692421]: Waiting for error group managing segments...
Jan 7 15:07:21.573623815 (791h38m21.44470686s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.574742956 (791h38m21.445825991s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.576831663 (791h38m21.447914708s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.577347465 (791h38m21.448430511s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.594180047 (791h38m21.465263092s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.596898944 (791h38m21.467981989s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.597996954 (791h38m21.46907999s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.598323697 (791h38m21.469406752s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.619734787 (791h38m21.490817822s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.655444514 (791h38m21.526527559s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.759546089 (791h38m21.630629124s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.032297580 (791h38m21.903380625s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694966023 (791h38m22.566049068s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694991682 (791h38m22.566074717s) [seg_pos={0 0}, upload=692421]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:07:22.695216660 (791h38m22.566299705s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695226038 (791h38m22.566309083s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.695609408 (791h38m22.566692463s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695618646 (791h38m22.566701681s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696042873 (791h38m22.567125918s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43498->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:07:22.696052972 (791h38m22.567136027s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696076738 (791h38m22.567159783s) [seg_pos={0 0}, upload=692421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:07:22.696162061 (791h38m22.567245106s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:22.696168693 (791h38m22.567251738s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:23.002248074 (791h38m22.87333112s) [upload=692421]: Done uploading segments...
Jan 7 15:07:23.002274104 (791h38m22.873357139s) [upload=692421]: Upload finished. err: <nil>
Duration: 6.119156195s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
16 out of 60 is failed (threshold=3) [X....XX..X..........XX..X....X.X...XX..X.X..XX........X.....] duration:
15.440579056s trace:
https://jaeger.dev.storj.io/trace/4aa072d0af3a78b5
FAIL (2026-01-07 15:12:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1h1gxgVjoHUkjKXA5xhTowGKi9q6wZGyzg8eqKgMHqsxF1ut5P, Piece ID: R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ) context deadline exceeded; piecestore: (Node ID: 1p8noFPJmn8RFi65iYw3wixTbJyfRNtwCzr8WK6Jehtiebkfi8, Piece ID: NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA) context deadline exceeded; piecestore: (Node ID: 1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, Piece ID: IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA) context deadline exceeded; piecestore: (Node ID: 1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, Piece ID: VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ) context deadline exceeded
======LOGS======
:07:28.569915240 (791h38m28.440998285s) [noise=true , address=77.90.31.10:4523 , piece_id=VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570208750 (791h38m28.441291785s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570372240 (791h38m28.441455285s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570789505 (791h38m28.44187254s) [noise=true , address=77.90.31.12:4598 , piece_id=3XJM4SSJ7TGON6T5OXQDPKE5PUBGAADTTR7H656IZ26LZADY42DA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570858336 (791h38m28.441941371s) [noise=true , address=45.91.116.14:4214 , piece_id=NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571261694 (791h38m28.442344739s) [noise=true , address=193.149.164.143:4696 , piece_id=75Z55LSF27EQ4BIP5ZJBRQC4WRMEO5OV4SD46K6OVUGB6EYQZYDQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571274067 (791h38m28.442357113s) [noise=true , address=145.79.227.44:4216 , piece_id=HBEEXUKULNI74JHM7SO2M5F6CX64CRYBPT7R5PUYYPJF7BK6H2JA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571644423 (791h38m28.442727468s) [noise=true , address=145.79.227.15:4895 , piece_id=ZJE23G6ZR7TWAK2H7BCSJA6H3KQ3U4VCLHHDQO6ET47KDKCU3FFA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571679760 (791h38m28.442762795s) [noise=true , address=145.79.227.32:4841 , piece_id=JMC54Q22ONLOFQLRUT2CFIVELVEC7F65RIYLJPPMUWKMMMRVFXBQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572067308 (791h38m28.443150343s) [noise=true , address=145.79.227.40:4076 , piece_id=VHXMJDLYLDLNVQMIKADA57ZN2XO6NJV5PBGQ76NG5BQGGMVTU7XA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572136760 (791h38m28.443219795s) [noise=true , address=64.49.13.155:4921 , piece_id=W7IBUJOAS3ZWSEBOL6FRGTNZAOIUUTM6HBUYVOEKGGPJRRLQBTPA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572408969 (791h38m28.443492004s) [noise=true , address=45.91.116.12:4252 , piece_id=IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572535730 (791h38m28.443618775s) [noise=true , address=145.79.227.34:4938 , piece_id=QHH24ERM3PCVUMNTT5MHTYZB3GV6N2GJQWVCQFP2LWV6TGF4BETA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572739538 (791h38m28.443822573s) [noise=true , address=64.49.13.169:4707 , piece_id=DWDI6QOGLDD37T27FMEPT5PX3QH75JZIE3L5WICCK5ZC3YXIZB5Q, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573009863 (791h38m28.444092898s) [noise=true , address=145.79.227.2:4132 , piece_id=KODYLQLGLPEIHU3N6A42EMD4GE3BVDUIMFV26IMHHZ5UJ4MBACLA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573075768 (791h38m28.444158814s) [noise=true , address=45.91.116.11:4141 , piece_id=R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573763268 (791h38m28.444846303s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:29.970413644 (791h38m29.841496679s) [upload=692426]: Next returned nil segment
Jan 7 15:07:29.970423102 (791h38m29.841506137s) [upload=692426]: Waiting for error group managing segments...
Jan 7 15:07:30.371305354 (791h38m30.242388399s) [noise=true , address=64.49.13.169:4707 , piece_id=DWDI6QOGLDD37T27FMEPT5PX3QH75JZIE3L5WICCK5ZC3YXIZB5Q, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.435851700 (791h38m30.306934745s) [noise=true , address=145.79.227.2:4132 , piece_id=KODYLQLGLPEIHU3N6A42EMD4GE3BVDUIMFV26IMHHZ5UJ4MBACLA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.438165455 (791h38m30.30924849s) [noise=true , address=145.79.227.40:4076 , piece_id=VHXMJDLYLDLNVQMIKADA57ZN2XO6NJV5PBGQ76NG5BQGGMVTU7XA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.446841345 (791h38m30.317924391s) [noise=true , address=145.79.227.15:4895 , piece_id=ZJE23G6ZR7TWAK2H7BCSJA6H3KQ3U4VCLHHDQO6ET47KDKCU3FFA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.498162610 (791h38m30.369245645s) [noise=true , address=145.79.227.44:4216 , piece_id=HBEEXUKULNI74JHM7SO2M5F6CX64CRYBPT7R5PUYYPJF7BK6H2JA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.583564478 (791h38m30.454647533s) [noise=true , address=145.79.227.34:4938 , piece_id=QHH24ERM3PCVUMNTT5MHTYZB3GV6N2GJQWVCQFP2LWV6TGF4BETA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.766796791 (791h38m30.637879826s) [noise=true , address=77.90.31.12:4598 , piece_id=3XJM4SSJ7TGON6T5OXQDPKE5PUBGAADTTR7H656IZ26LZADY42DA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.856815360 (791h38m30.727898395s) [noise=true , address=45.91.116.12:4252 , piece_id=IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.866531521 (791h38m30.737614556s) [noise=true , address=45.91.116.14:4214 , piece_id=NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.961126103 (791h38m30.832209148s) [noise=true , address=45.91.116.11:4141 , piece_id=R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.252513510 (791h38m31.123596545s) [noise=true , address=193.149.164.143:4696 , piece_id=75Z55LSF27EQ4BIP5ZJBRQC4WRMEO5OV4SD46K6OVUGB6EYQZYDQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.420253843 (791h38m31.291336888s) [noise=true , address=77.90.31.10:4523 , piece_id=VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.646183618 (791h38m31.517266693s) [noise=true , address=64.49.13.155:4921 , piece_id=W7IBUJOAS3ZWSEBOL6FRGTNZAOIUUTM6HBUYVOEKGGPJRRLQBTPA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.752165924 (791h38m31.623248969s) [noise=true , address=145.79.227.32:4841 , piece_id=JMC54Q22ONLOFQLRUT2CFIVELVEC7F65RIYLJPPMUWKMMMRVFXBQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.752196092 (791h38m31.623279137s) [seg_pos={0 0}, upload=692426]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:07:31.752531661 (791h38m31.623614706s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752540307 (791h38m31.623623352s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752588209 (791h38m31.623671254s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QucqyXk8Zx4q4uK6nMULHHRjZCRfQDoUREoHZNEjsQwVQt7gK): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752612595 (791h38m31.62369567s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752785755 (791h38m31.6238688s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752793991 (791h38m31.623877046s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752946962 (791h38m31.624030007s) [noise=true , address=45.91.116.9:4273 , piece_id=Q3WO7HBOHJWVUZUWNJ4FZN26PEB4IHHNFAMWVYBQTN3QCPGVTRHQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752954967 (791h38m31.624038002s) [noise=true , address=45.91.116.9:4273 , piece_id=Q3WO7HBOHJWVUZUWNJ4FZN26PEB4IHHNFAMWVYBQTN3QCPGVTRHQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752972330 (791h38m31.624055376s) [seg_pos={0 0}, upload=692426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:07:31.753067240 (791h38m31.624150275s) [upload=692426]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:31.753072662 (791h38m31.624155707s) [upload=692426]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:32.044442855 (791h38m31.9155259s) [upload=692426]: Done uploading segments...
Jan 7 15:07:32.044467151 (791h38m31.915550196s) [upload=692426]: Upload finished. err: <nil>
Duration: 15.440608933s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
36 out of 60 is failed (threshold=3) [.XXXXXXX.XXXXXXXXX..XX.X.XXXX....X....XXX..X..XXX..X...XXX.X] duration:
1m0.000871482s trace:
https://jaeger.dev.storj.io/trace/41ad92cdda135526
FAIL (2026-01-07 15:12:15 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000883887s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
3.761219923s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:12:11 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 3.761249728s
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [.......X........X.............X......XXX..X..X...X......X...] duration:
9.548660355s trace:
https://jaeger.dev.storj.io/trace/23514c16b68dbbaa
FAIL (2026-01-07 15:12:05 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 9.548691196s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000793158s trace:
https://jaeger.dev.storj.io/trace/6ce5291da55a093b
FAIL (2026-01-07 15:11:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000816603s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [X.......X........X.............X......XXX..X..X...X......X..] duration:
19.206210398s trace:
https://jaeger.dev.storj.io/trace/23514c16b68dbbaa
FAIL (2026-01-07 15:11:52 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 19.206289859s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
1.791629568s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:11:51 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 1.791659988s
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
50 out of 60 is failed (threshold=3) [.X.X.XXXXX.XX....XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.] duration:
24.52384201s trace:
https://jaeger.dev.storj.io/trace/4e355d22f94f9de
FAIL (2026-01-07 15:11:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.523877258s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X....XX..X..........XX..X....X.X...XX..X.X..XX........X....] duration:
15.432749324s trace:
https://jaeger.dev.storj.io/trace/4aa072d0af3a78b5
FAIL (2026-01-07 15:11:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1h1gxgVjoHUkjKXA5xhTowGKi9q6wZGyzg8eqKgMHqsxF1ut5P, Piece ID: R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ) context deadline exceeded; piecestore: (Node ID: 1p8noFPJmn8RFi65iYw3wixTbJyfRNtwCzr8WK6Jehtiebkfi8, Piece ID: NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA) context deadline exceeded; piecestore: (Node ID: 1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, Piece ID: IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA) context deadline exceeded; piecestore: (Node ID: 1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, Piece ID: VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ) context deadline exceeded
======LOGS======
:07:28.569915240 (791h38m28.440998285s) [noise=true , address=77.90.31.10:4523 , piece_id=VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570208750 (791h38m28.441291785s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570372240 (791h38m28.441455285s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570789505 (791h38m28.44187254s) [noise=true , address=77.90.31.12:4598 , piece_id=3XJM4SSJ7TGON6T5OXQDPKE5PUBGAADTTR7H656IZ26LZADY42DA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570858336 (791h38m28.441941371s) [noise=true , address=45.91.116.14:4214 , piece_id=NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571261694 (791h38m28.442344739s) [noise=true , address=193.149.164.143:4696 , piece_id=75Z55LSF27EQ4BIP5ZJBRQC4WRMEO5OV4SD46K6OVUGB6EYQZYDQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571274067 (791h38m28.442357113s) [noise=true , address=145.79.227.44:4216 , piece_id=HBEEXUKULNI74JHM7SO2M5F6CX64CRYBPT7R5PUYYPJF7BK6H2JA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571644423 (791h38m28.442727468s) [noise=true , address=145.79.227.15:4895 , piece_id=ZJE23G6ZR7TWAK2H7BCSJA6H3KQ3U4VCLHHDQO6ET47KDKCU3FFA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571679760 (791h38m28.442762795s) [noise=true , address=145.79.227.32:4841 , piece_id=JMC54Q22ONLOFQLRUT2CFIVELVEC7F65RIYLJPPMUWKMMMRVFXBQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572067308 (791h38m28.443150343s) [noise=true , address=145.79.227.40:4076 , piece_id=VHXMJDLYLDLNVQMIKADA57ZN2XO6NJV5PBGQ76NG5BQGGMVTU7XA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572136760 (791h38m28.443219795s) [noise=true , address=64.49.13.155:4921 , piece_id=W7IBUJOAS3ZWSEBOL6FRGTNZAOIUUTM6HBUYVOEKGGPJRRLQBTPA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572408969 (791h38m28.443492004s) [noise=true , address=45.91.116.12:4252 , piece_id=IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572535730 (791h38m28.443618775s) [noise=true , address=145.79.227.34:4938 , piece_id=QHH24ERM3PCVUMNTT5MHTYZB3GV6N2GJQWVCQFP2LWV6TGF4BETA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572739538 (791h38m28.443822573s) [noise=true , address=64.49.13.169:4707 , piece_id=DWDI6QOGLDD37T27FMEPT5PX3QH75JZIE3L5WICCK5ZC3YXIZB5Q, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573009863 (791h38m28.444092898s) [noise=true , address=145.79.227.2:4132 , piece_id=KODYLQLGLPEIHU3N6A42EMD4GE3BVDUIMFV26IMHHZ5UJ4MBACLA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573075768 (791h38m28.444158814s) [noise=true , address=45.91.116.11:4141 , piece_id=R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573763268 (791h38m28.444846303s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:29.970413644 (791h38m29.841496679s) [upload=692426]: Next returned nil segment
Jan 7 15:07:29.970423102 (791h38m29.841506137s) [upload=692426]: Waiting for error group managing segments...
Jan 7 15:07:30.371305354 (791h38m30.242388399s) [noise=true , address=64.49.13.169:4707 , piece_id=DWDI6QOGLDD37T27FMEPT5PX3QH75JZIE3L5WICCK5ZC3YXIZB5Q, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.435851700 (791h38m30.306934745s) [noise=true , address=145.79.227.2:4132 , piece_id=KODYLQLGLPEIHU3N6A42EMD4GE3BVDUIMFV26IMHHZ5UJ4MBACLA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.438165455 (791h38m30.30924849s) [noise=true , address=145.79.227.40:4076 , piece_id=VHXMJDLYLDLNVQMIKADA57ZN2XO6NJV5PBGQ76NG5BQGGMVTU7XA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.446841345 (791h38m30.317924391s) [noise=true , address=145.79.227.15:4895 , piece_id=ZJE23G6ZR7TWAK2H7BCSJA6H3KQ3U4VCLHHDQO6ET47KDKCU3FFA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.498162610 (791h38m30.369245645s) [noise=true , address=145.79.227.44:4216 , piece_id=HBEEXUKULNI74JHM7SO2M5F6CX64CRYBPT7R5PUYYPJF7BK6H2JA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.583564478 (791h38m30.454647533s) [noise=true , address=145.79.227.34:4938 , piece_id=QHH24ERM3PCVUMNTT5MHTYZB3GV6N2GJQWVCQFP2LWV6TGF4BETA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.766796791 (791h38m30.637879826s) [noise=true , address=77.90.31.12:4598 , piece_id=3XJM4SSJ7TGON6T5OXQDPKE5PUBGAADTTR7H656IZ26LZADY42DA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.856815360 (791h38m30.727898395s) [noise=true , address=45.91.116.12:4252 , piece_id=IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.866531521 (791h38m30.737614556s) [noise=true , address=45.91.116.14:4214 , piece_id=NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.961126103 (791h38m30.832209148s) [noise=true , address=45.91.116.11:4141 , piece_id=R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.252513510 (791h38m31.123596545s) [noise=true , address=193.149.164.143:4696 , piece_id=75Z55LSF27EQ4BIP5ZJBRQC4WRMEO5OV4SD46K6OVUGB6EYQZYDQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.420253843 (791h38m31.291336888s) [noise=true , address=77.90.31.10:4523 , piece_id=VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.646183618 (791h38m31.517266693s) [noise=true , address=64.49.13.155:4921 , piece_id=W7IBUJOAS3ZWSEBOL6FRGTNZAOIUUTM6HBUYVOEKGGPJRRLQBTPA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.752165924 (791h38m31.623248969s) [noise=true , address=145.79.227.32:4841 , piece_id=JMC54Q22ONLOFQLRUT2CFIVELVEC7F65RIYLJPPMUWKMMMRVFXBQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.752196092 (791h38m31.623279137s) [seg_pos={0 0}, upload=692426]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:07:31.752531661 (791h38m31.623614706s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752540307 (791h38m31.623623352s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752588209 (791h38m31.623671254s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QucqyXk8Zx4q4uK6nMULHHRjZCRfQDoUREoHZNEjsQwVQt7gK): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752612595 (791h38m31.62369567s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752785755 (791h38m31.6238688s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752793991 (791h38m31.623877046s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752946962 (791h38m31.624030007s) [noise=true , address=45.91.116.9:4273 , piece_id=Q3WO7HBOHJWVUZUWNJ4FZN26PEB4IHHNFAMWVYBQTN3QCPGVTRHQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752954967 (791h38m31.624038002s) [noise=true , address=45.91.116.9:4273 , piece_id=Q3WO7HBOHJWVUZUWNJ4FZN26PEB4IHHNFAMWVYBQTN3QCPGVTRHQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752972330 (791h38m31.624055376s) [seg_pos={0 0}, upload=692426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:07:31.753067240 (791h38m31.624150275s) [upload=692426]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:31.753072662 (791h38m31.624155707s) [upload=692426]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:32.044442855 (791h38m31.9155259s) [upload=692426]: Done uploading segments...
Jan 7 15:07:32.044467151 (791h38m31.915550196s) [upload=692426]: Upload finished. err: <nil>
Duration: 15.432781265s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
1.798335387s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:11:21 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 1.798364552s
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X.X....X.........X.X..........X..X..X.XX.............X......] duration:
6.282483655s trace:
https://jaeger.dev.storj.io/trace/1f3c53a17b6f7fb6
FAIL (2026-01-07 15:11:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA) context deadline exceeded
======LOGS======
Jan 7 15:07:19.312082232 (791h38m19.183165267s) [upload=692421]: Starting upload
Jan 7 15:07:19.312207981 (791h38m19.183291046s) [upload=692421]: Done starting upload
Jan 7 15:07:19.312284578 (791h38m19.183367643s) [upload=692421]: Uploading segments...
Jan 7 15:07:19.312502482 (791h38m19.183585527s) [upload=692421]: Got next segment. Inline: false
Jan 7 15:07:19.312544662 (791h38m19.183627717s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:07:19.312550694 (791h38m19.183633739s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:07:19.569325994 (791h38m19.440409039s) [seg_pos={0 0}, upload=692421]: Begin upload segment...
Jan 7 15:07:19.569382582 (791h38m19.440465627s) [seg_pos={0 0}, upload=692421]: Done begin upload segment.
Jan 7 15:07:19.569492361 (791h38m19.440575406s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570231268 (791h38m19.441314313s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570501242 (791h38m19.441584277s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570740167 (791h38m19.441823202s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571014419 (791h38m19.442097464s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571272691 (791h38m19.442355736s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571546643 (791h38m19.442629688s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571846985 (791h38m19.44293002s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572147166 (791h38m19.443230201s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572390539 (791h38m19.443473584s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572632821 (791h38m19.443715856s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572895762 (791h38m19.443978797s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573185794 (791h38m19.444268829s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573558714 (791h38m19.444641749s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583179604 (791h38m19.454262639s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583602939 (791h38m19.454685975s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:21.192726317 (791h38m21.063809362s) [upload=692421]: Next returned nil segment
Jan 7 15:07:21.192743761 (791h38m21.063826806s) [upload=692421]: Waiting for error group managing segments...
Jan 7 15:07:21.573623815 (791h38m21.44470686s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.574742956 (791h38m21.445825991s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.576831663 (791h38m21.447914708s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.577347465 (791h38m21.448430511s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.594180047 (791h38m21.465263092s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.596898944 (791h38m21.467981989s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.597996954 (791h38m21.46907999s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.598323697 (791h38m21.469406752s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.619734787 (791h38m21.490817822s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.655444514 (791h38m21.526527559s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.759546089 (791h38m21.630629124s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.032297580 (791h38m21.903380625s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694966023 (791h38m22.566049068s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694991682 (791h38m22.566074717s) [seg_pos={0 0}, upload=692421]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:07:22.695216660 (791h38m22.566299705s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695226038 (791h38m22.566309083s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.695609408 (791h38m22.566692463s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695618646 (791h38m22.566701681s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696042873 (791h38m22.567125918s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43498->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:07:22.696052972 (791h38m22.567136027s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696076738 (791h38m22.567159783s) [seg_pos={0 0}, upload=692421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:07:22.696162061 (791h38m22.567245106s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:22.696168693 (791h38m22.567251738s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:23.002248074 (791h38m22.87333112s) [upload=692421]: Done uploading segments...
Jan 7 15:07:23.002274104 (791h38m22.873357139s) [upload=692421]: Upload finished. err: <nil>
Duration: 6.282518301s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
36 out of 60 is failed (threshold=3) [X.XXXXXXX.XXXXXXXXX..XX.X.XXXX....X....XXX..X..XXX..X...XXX.] duration:
19.404136435s trace:
https://jaeger.dev.storj.io/trace/9f53c6c012893e8
FAIL (2026-01-07 15:11:15 UTC)
orbiter warning: context deadline exceeded
Duration: 19.404153517s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
2.88304973s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:11:04 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 2.883084958s
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [.X.......X........X.............X......XXX..X..X...X......X.] duration:
13.786085864s trace:
https://jaeger.dev.storj.io/trace/23514c16b68dbbaa
FAIL (2026-01-07 15:11:01 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 13.786108266s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000952501s trace:
https://jaeger.dev.storj.io/trace/3340cf5ea5e30e18
FAIL (2026-01-07 15:10:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00097747s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
37 out of 60 is failed (threshold=3) [XX.XXXXXXX.XXXXXXXXX..XX.X.XXXX....X....XXX..X..XXX..X...XXX] duration:
1m0.001378226s trace:
https://jaeger.dev.storj.io/trace/9f53c6c012893e8
FAIL (2026-01-07 15:10:55 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001392714s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X.X....X.........X.X..........X..X..X.XX.............X.....] duration:
5.755493113s trace:
https://jaeger.dev.storj.io/trace/1f3c53a17b6f7fb6
FAIL (2026-01-07 15:10:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA) context deadline exceeded
======LOGS======
Jan 7 15:07:19.312082232 (791h38m19.183165267s) [upload=692421]: Starting upload
Jan 7 15:07:19.312207981 (791h38m19.183291046s) [upload=692421]: Done starting upload
Jan 7 15:07:19.312284578 (791h38m19.183367643s) [upload=692421]: Uploading segments...
Jan 7 15:07:19.312502482 (791h38m19.183585527s) [upload=692421]: Got next segment. Inline: false
Jan 7 15:07:19.312544662 (791h38m19.183627717s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 15:07:19.312550694 (791h38m19.183633739s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 15:07:19.569325994 (791h38m19.440409039s) [seg_pos={0 0}, upload=692421]: Begin upload segment...
Jan 7 15:07:19.569382582 (791h38m19.440465627s) [seg_pos={0 0}, upload=692421]: Done begin upload segment.
Jan 7 15:07:19.569492361 (791h38m19.440575406s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570231268 (791h38m19.441314313s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570501242 (791h38m19.441584277s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.570740167 (791h38m19.441823202s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571014419 (791h38m19.442097464s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571272691 (791h38m19.442355736s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571546643 (791h38m19.442629688s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.571846985 (791h38m19.44293002s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572147166 (791h38m19.443230201s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572390539 (791h38m19.443473584s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572632821 (791h38m19.443715856s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.572895762 (791h38m19.443978797s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573185794 (791h38m19.444268829s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.573558714 (791h38m19.444641749s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583179604 (791h38m19.454262639s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:19.583602939 (791h38m19.454685975s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Uploading piece...
Jan 7 15:07:21.192726317 (791h38m21.063809362s) [upload=692421]: Next returned nil segment
Jan 7 15:07:21.192743761 (791h38m21.063826806s) [upload=692421]: Waiting for error group managing segments...
Jan 7 15:07:21.573623815 (791h38m21.44470686s) [noise=true , address=145.79.227.39:4014 , piece_id=TQLYFKCMTA2TWPD2PL3VWB46PY35GXRF5SGYXPCP5OD2LRNH7E6A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.574742956 (791h38m21.445825991s) [noise=true , address=145.79.227.7:4295 , piece_id=FLUOWKZVBOCLN7T32GME3JVWX3DECVJAZCIL7KWQPWCU4B7XLJNQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.576831663 (791h38m21.447914708s) [noise=true , address=145.79.227.12:4765 , piece_id=NPKI6YWGZHQIYENKHN2EPGBKLC55MD6DDDWL3JFJ6AGLQVHOZBJQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.577347465 (791h38m21.448430511s) [noise=true , address=145.79.227.5:4223 , piece_id=SGAFBAXAYLNGT34O5ZXVFLPGVUGVGRHGYDHZNRK5YMVAZCMAG7CQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.594180047 (791h38m21.465263092s) [noise=true , address=145.79.227.3:4149 , piece_id=YCUUC5Y5HD5WIEISVS5LTMYL6EVLV5YZPAOHXI7SFFAPOFMNMFUQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.596898944 (791h38m21.467981989s) [noise=true , address=145.79.227.8:4331 , piece_id=OYCYYIVJU6NN527I7BROT3XBHGZMCEK35OAL43JRQDGKS76TBOZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.597996954 (791h38m21.46907999s) [noise=true , address=145.79.227.14:4840 , piece_id=JHQO7YUQZ34K7OGFAO7VODQQBAF3XUDCTLHCSG2VOK4N7O4FRCVQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.598323697 (791h38m21.469406752s) [noise=true , address=145.79.227.48:4302 , piece_id=5IDDJRNV6FGYWOCJSGHBSHDM7FFKHRBJY6WT7Q4W2C5BGUSAFH3A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.619734787 (791h38m21.490817822s) [noise=true , address=145.79.227.38:4977 , piece_id=R3LHEHRTLFL3XR75ZUAH37L3RG662DGSYDIB6INTBHS46AJTFJZA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.655444514 (791h38m21.526527559s) [noise=true , address=145.79.227.40:4049 , piece_id=PF7NEP3EF6UAFVZVKMLVGFUWBQBHD6E44C3EJ6EOJQN5BLLDFA5A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:21.759546089 (791h38m21.630629124s) [noise=true , address=145.79.227.47:4266 , piece_id=3XRAEFGRMUFSVDOWHUT3S7JUPJH44CMCBLO2VDWKDA3BTYAEKGYA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.032297580 (791h38m21.903380625s) [noise=true , address=45.91.116.9:4261 , piece_id=IF6JVFEWXETSEK5JRSW4Z4PUZW4NLBN3ZP5LUG4BLMXEOXCATRSA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694966023 (791h38m22.566049068s) [noise=true , address=77.90.31.21:4880 , piece_id=7I3FV7NRHYOIDCEXD3II5U2LPFIW3TIDUISC46PNJUJTQNY3GHXA, seg_pos={0 0}, upload=692421]: Done uploading piece. err: <nil>
Jan 7 15:07:22.694991682 (791h38m22.566074717s) [seg_pos={0 0}, upload=692421]: Segment reached optimal threshold of 13 pieces.
Jan 7 15:07:22.695216660 (791h38m22.566299705s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695226038 (791h38m22.566309083s) [noise=true , address=45.91.116.14:4192 , piece_id=X2UTBZ55U7EGFZST5VA2PELZAGKQNFMX4NUA7ORSBVYFV4TAJAQQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.695609408 (791h38m22.566692463s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:22.695618646 (791h38m22.566701681s) [noise=true , address=45.91.116.12:4227 , piece_id=YUU7PT3GET7STFEG4NF7RMRYNW2MHURQP7WO4KFIHZPAP75QNA4A, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696042873 (791h38m22.567125918s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43498->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 15:07:22.696052972 (791h38m22.567136027s) [noise=true , address=45.91.116.5:4298 , piece_id=K5PTZT3RDIZ5S4WICPCGB5IFWYJH7BNOEPBQ62R5KZLMEETBY3QQ, seg_pos={0 0}, upload=692421]: Long tail cancellation due to optimal threshold
Jan 7 15:07:22.696076738 (791h38m22.567159783s) [seg_pos={0 0}, upload=692421]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 15:07:22.696162061 (791h38m22.567245106s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:22.696168693 (791h38m22.567251738s) [upload=692421]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:23.002248074 (791h38m22.87333112s) [upload=692421]: Done uploading segments...
Jan 7 15:07:23.002274104 (791h38m22.873357139s) [upload=692421]: Upload finished. err: <nil>
Duration: 5.755520515s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [..X.......X........X.............X......XXX..X..X...X......X] duration:
30.000859484s trace:
https://jaeger.dev.storj.io/trace/23514c16b68dbbaa
FAIL (2026-01-07 15:10:41 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 30.000887506s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
3.648185225s remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace:
https://jaeger.dev.storj.io/trace/3bcf4e48baa6dd71
FAIL (2026-01-07 15:10:40 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 3.648225102s
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.X....XX..X..........XX..X....X.X...XX..X.X..XX........X...] duration:
15.273546225s trace:
https://jaeger.dev.storj.io/trace/4aa072d0af3a78b5
FAIL (2026-01-07 15:10:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1h1gxgVjoHUkjKXA5xhTowGKi9q6wZGyzg8eqKgMHqsxF1ut5P, Piece ID: R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ) context deadline exceeded; piecestore: (Node ID: 1p8noFPJmn8RFi65iYw3wixTbJyfRNtwCzr8WK6Jehtiebkfi8, Piece ID: NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA) context deadline exceeded; piecestore: (Node ID: 1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, Piece ID: IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA) context deadline exceeded; piecestore: (Node ID: 1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, Piece ID: VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ) context deadline exceeded
======LOGS======
:07:28.569915240 (791h38m28.440998285s) [noise=true , address=77.90.31.10:4523 , piece_id=VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570208750 (791h38m28.441291785s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570372240 (791h38m28.441455285s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570789505 (791h38m28.44187254s) [noise=true , address=77.90.31.12:4598 , piece_id=3XJM4SSJ7TGON6T5OXQDPKE5PUBGAADTTR7H656IZ26LZADY42DA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.570858336 (791h38m28.441941371s) [noise=true , address=45.91.116.14:4214 , piece_id=NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571261694 (791h38m28.442344739s) [noise=true , address=193.149.164.143:4696 , piece_id=75Z55LSF27EQ4BIP5ZJBRQC4WRMEO5OV4SD46K6OVUGB6EYQZYDQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571274067 (791h38m28.442357113s) [noise=true , address=145.79.227.44:4216 , piece_id=HBEEXUKULNI74JHM7SO2M5F6CX64CRYBPT7R5PUYYPJF7BK6H2JA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571644423 (791h38m28.442727468s) [noise=true , address=145.79.227.15:4895 , piece_id=ZJE23G6ZR7TWAK2H7BCSJA6H3KQ3U4VCLHHDQO6ET47KDKCU3FFA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.571679760 (791h38m28.442762795s) [noise=true , address=145.79.227.32:4841 , piece_id=JMC54Q22ONLOFQLRUT2CFIVELVEC7F65RIYLJPPMUWKMMMRVFXBQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572067308 (791h38m28.443150343s) [noise=true , address=145.79.227.40:4076 , piece_id=VHXMJDLYLDLNVQMIKADA57ZN2XO6NJV5PBGQ76NG5BQGGMVTU7XA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572136760 (791h38m28.443219795s) [noise=true , address=64.49.13.155:4921 , piece_id=W7IBUJOAS3ZWSEBOL6FRGTNZAOIUUTM6HBUYVOEKGGPJRRLQBTPA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572408969 (791h38m28.443492004s) [noise=true , address=45.91.116.12:4252 , piece_id=IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572535730 (791h38m28.443618775s) [noise=true , address=145.79.227.34:4938 , piece_id=QHH24ERM3PCVUMNTT5MHTYZB3GV6N2GJQWVCQFP2LWV6TGF4BETA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.572739538 (791h38m28.443822573s) [noise=true , address=64.49.13.169:4707 , piece_id=DWDI6QOGLDD37T27FMEPT5PX3QH75JZIE3L5WICCK5ZC3YXIZB5Q, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573009863 (791h38m28.444092898s) [noise=true , address=145.79.227.2:4132 , piece_id=KODYLQLGLPEIHU3N6A42EMD4GE3BVDUIMFV26IMHHZ5UJ4MBACLA, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573075768 (791h38m28.444158814s) [noise=true , address=45.91.116.11:4141 , piece_id=R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:28.573763268 (791h38m28.444846303s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Uploading piece...
Jan 7 15:07:29.970413644 (791h38m29.841496679s) [upload=692426]: Next returned nil segment
Jan 7 15:07:29.970423102 (791h38m29.841506137s) [upload=692426]: Waiting for error group managing segments...
Jan 7 15:07:30.371305354 (791h38m30.242388399s) [noise=true , address=64.49.13.169:4707 , piece_id=DWDI6QOGLDD37T27FMEPT5PX3QH75JZIE3L5WICCK5ZC3YXIZB5Q, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.435851700 (791h38m30.306934745s) [noise=true , address=145.79.227.2:4132 , piece_id=KODYLQLGLPEIHU3N6A42EMD4GE3BVDUIMFV26IMHHZ5UJ4MBACLA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.438165455 (791h38m30.30924849s) [noise=true , address=145.79.227.40:4076 , piece_id=VHXMJDLYLDLNVQMIKADA57ZN2XO6NJV5PBGQ76NG5BQGGMVTU7XA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.446841345 (791h38m30.317924391s) [noise=true , address=145.79.227.15:4895 , piece_id=ZJE23G6ZR7TWAK2H7BCSJA6H3KQ3U4VCLHHDQO6ET47KDKCU3FFA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.498162610 (791h38m30.369245645s) [noise=true , address=145.79.227.44:4216 , piece_id=HBEEXUKULNI74JHM7SO2M5F6CX64CRYBPT7R5PUYYPJF7BK6H2JA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.583564478 (791h38m30.454647533s) [noise=true , address=145.79.227.34:4938 , piece_id=QHH24ERM3PCVUMNTT5MHTYZB3GV6N2GJQWVCQFP2LWV6TGF4BETA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.766796791 (791h38m30.637879826s) [noise=true , address=77.90.31.12:4598 , piece_id=3XJM4SSJ7TGON6T5OXQDPKE5PUBGAADTTR7H656IZ26LZADY42DA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.856815360 (791h38m30.727898395s) [noise=true , address=45.91.116.12:4252 , piece_id=IXOVF6ODXSLELP3DOMYSK7VE4DHVYLGOOJBOGPXHVL27WNPJJBRA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.866531521 (791h38m30.737614556s) [noise=true , address=45.91.116.14:4214 , piece_id=NG5O7RCFSHI5YIF4RZ3FMMCRQUNAWSF7JJAYGYT5SRTOYNSEABCA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:30.961126103 (791h38m30.832209148s) [noise=true , address=45.91.116.11:4141 , piece_id=R7XNXBU5SP7SHFYDKVSB23XWI2COP3BMAXXWLRYZ4FV3ITM6XSLQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.252513510 (791h38m31.123596545s) [noise=true , address=193.149.164.143:4696 , piece_id=75Z55LSF27EQ4BIP5ZJBRQC4WRMEO5OV4SD46K6OVUGB6EYQZYDQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.420253843 (791h38m31.291336888s) [noise=true , address=77.90.31.10:4523 , piece_id=VQE4Y7F7PSI6F7UA2N74O7CZMW4OQZNDZVQVRRMQJMJ2YQEGX3PQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.646183618 (791h38m31.517266693s) [noise=true , address=64.49.13.155:4921 , piece_id=W7IBUJOAS3ZWSEBOL6FRGTNZAOIUUTM6HBUYVOEKGGPJRRLQBTPA, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.752165924 (791h38m31.623248969s) [noise=true , address=145.79.227.32:4841 , piece_id=JMC54Q22ONLOFQLRUT2CFIVELVEC7F65RIYLJPPMUWKMMMRVFXBQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: <nil>
Jan 7 15:07:31.752196092 (791h38m31.623279137s) [seg_pos={0 0}, upload=692426]: Segment reached optimal threshold of 14 pieces.
Jan 7 15:07:31.752531661 (791h38m31.623614706s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752540307 (791h38m31.623623352s) [noise=true , address=77.90.31.11:4586 , piece_id=SOCOBRKCKAERZTESY3Y632URTIIKQBESZT2LBC5SX5ECEPXE4SLQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752588209 (791h38m31.623671254s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QucqyXk8Zx4q4uK6nMULHHRjZCRfQDoUREoHZNEjsQwVQt7gK): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752612595 (791h38m31.62369567s) [noise=true , address=145.79.227.29:4952 , piece_id=VGUVORNSDKH5PEBZ2A3T2QA2OE2BIPAFNTTUZ3D7XJ7CKFVX7B5A, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752785755 (791h38m31.6238688s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752793991 (791h38m31.623877046s) [noise=true , address=145.79.227.49:4359 , piece_id=P6HLDUFRSEEMITEJJRM5EALEOGJKGB6ELVRJYT6YX372BTH5GABQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752946962 (791h38m31.624030007s) [noise=true , address=45.91.116.9:4273 , piece_id=Q3WO7HBOHJWVUZUWNJ4FZN26PEB4IHHNFAMWVYBQTN3QCPGVTRHQ, seg_pos={0 0}, upload=692426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf): protocol: expected piece hash; context canceled; EOF
Jan 7 15:07:31.752954967 (791h38m31.624038002s) [noise=true , address=45.91.116.9:4273 , piece_id=Q3WO7HBOHJWVUZUWNJ4FZN26PEB4IHHNFAMWVYBQTN3QCPGVTRHQ, seg_pos={0 0}, upload=692426]: Long tail cancellation due to optimal threshold
Jan 7 15:07:31.752972330 (791h38m31.624055376s) [seg_pos={0 0}, upload=692426]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 15:07:31.753067240 (791h38m31.624150275s) [upload=692426]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 15:07:31.753072662 (791h38m31.624155707s) [upload=692426]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 15:07:32.044442855 (791h38m31.9155259s) [upload=692426]: Done uploading segments...
Jan 7 15:07:32.044467151 (791h38m31.915550196s) [upload=692426]: Upload finished. err: <nil>
Duration: 15.273563367s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
51 out of 60 is failed (threshold=3) [X.X.X.XXXXX.XX....XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration:
30.001305575s trace:
https://jaeger.dev.storj.io/trace/4e355d22f94f9de
FAIL (2026-01-07 15:10:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001328339s
edge
gateway
us-select-1
|