Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXXXXXXX.XX] duration:
5.001685409s trace:
https://jaeger.dev.storj.io/trace/2b6bf858f09667dd
FAIL (2026-01-11 16:08:34 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001708653s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
35 out of 60 is failed (threshold=3) [.XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X....XX..X] duration:
30.000874993s trace:
https://jaeger.dev.storj.io/trace/5fd65bf611330ee9
FAIL (2026-01-11 16:08:33 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000890133s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [X.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXXXXXXX.X] duration:
5.000966044s trace:
https://jaeger.dev.storj.io/trace/94be91e80eea875
FAIL (2026-01-11 16:08:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000988106s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [..XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X....XX..] duration:
22.833885003s trace:
https://jaeger.dev.storj.io/trace/4794a26b94d78763
FAIL (2026-01-11 16:08:03 UTC)
orbiter warning: context deadline exceeded
Duration: 22.833912476s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [......X........X......X.X...........X..........X...........X] duration:
1m0.000919871s trace:
https://jaeger.dev.storj.io/trace/1f25644248521ec3
FAIL (2026-01-11 16:07:55 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.00094044s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXXXXXXX.] duration:
3.86727434s trace:
https://jaeger.dev.storj.io/trace/6635bcaa886f5cd3
FAIL (2026-01-11 16:07:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.867298496s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [...XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X....XX.] duration:
28.732675885s trace:
https://jaeger.dev.storj.io/trace/4794a26b94d78763
FAIL (2026-01-11 16:07:02 UTC)
orbiter warning: context deadline exceeded
Duration: 28.732703177s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.......X........X......X.X...........X..........X...........] duration:
16.567128353s trace:
https://jaeger.dev.storj.io/trace/bb0bc83ebd819e9
FAIL (2026-01-11 16:06:42 UTC)
orbiter warning: context deadline exceeded
Duration: 16.567153831s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [....XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X....XX] duration:
30.000952466s trace:
https://jaeger.dev.storj.io/trace/4794a26b94d78763
FAIL (2026-01-11 16:06:14 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000979017s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXXXXXXX] duration:
5.001069834s trace:
https://jaeger.dev.storj.io/trace/6635bcaa886f5cd3
FAIL (2026-01-11 16:06:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001093307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [........X........X......X.X...........X..........X..........] duration:
13.156877812s trace:
https://jaeger.dev.storj.io/trace/bb0bc83ebd819e9
FAIL (2026-01-11 16:05:47 UTC)
orbiter warning: context deadline exceeded
Duration: 13.156900786s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.554560789s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 16:05:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 5.554592169s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXXXXXX] duration:
5.001558656s trace:
https://jaeger.dev.storj.io/trace/2d7d457e9bc357b4
FAIL (2026-01-11 16:05:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.0015928s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [.....XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X....X] duration:
30.001207502s trace:
https://jaeger.dev.storj.io/trace/5ea97b36a9caa790
FAIL (2026-01-11 16:05:16 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001241024s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.........X........X......X.X...........X..........X.........] duration:
15.75533648s trace:
https://jaeger.dev.storj.io/trace/bb0bc83ebd819e9
FAIL (2026-01-11 16:05:01 UTC)
orbiter warning: context deadline exceeded
Duration: 15.755380994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXXXXX] duration:
5.000949505s trace:
https://jaeger.dev.storj.io/trace/6187fc41e73ca633
FAIL (2026-01-11 16:04:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00099482s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.97275398s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 16:04:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 5.972780761s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.187225566s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 16:04:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 6.187251376s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..........X........X......X.X...........X..........X........] duration:
26.705804839s trace:
https://jaeger.dev.storj.io/trace/bb0bc83ebd819e9
FAIL (2026-01-11 16:04:11 UTC)
orbiter warning: context deadline exceeded
Duration: 26.705823845s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [......XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X....] duration:
26.37077238s trace:
https://jaeger.dev.storj.io/trace/2760d69b475b327
FAIL (2026-01-11 16:04:05 UTC)
orbiter warning: context deadline exceeded
Duration: 26.3707993s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [.XXXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXXXX] duration:
5.000753333s trace:
https://jaeger.dev.storj.io/trace/506abe6ea733290b
FAIL (2026-01-11 16:04:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000817093s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [X......XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X...] duration:
23.319888542s trace:
https://jaeger.dev.storj.io/trace/2760d69b475b327
FAIL (2026-01-11 16:03:38 UTC)
orbiter warning: context deadline exceeded
Duration: 23.319914852s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.814289864s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 16:03:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 5.814308861s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...........X........X......X.X...........X..........X.......] duration:
22.844671218s trace:
https://jaeger.dev.storj.io/trace/bb0bc83ebd819e9
FAIL (2026-01-11 16:03:27 UTC)
orbiter warning: context deadline exceeded
Duration: 22.844691907s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [X.XXXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXXX] duration:
5.001838378s trace:
https://jaeger.dev.storj.io/trace/2c8a23d4a514d2b8
FAIL (2026-01-11 16:02:59 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001861482s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [XX......XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X..] duration:
29.716745333s trace:
https://jaeger.dev.storj.io/trace/2760d69b475b327
FAIL (2026-01-11 16:02:57 UTC)
orbiter warning: context deadline exceeded
Duration: 29.716773969s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.675175223s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 16:02:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 5.675203537s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.762905901s trace:
https://jaeger.dev.storj.io/trace/15f47b4548974807
FAIL (2026-01-11 16:02:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload
Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload
Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments...
Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false
Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment...
Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment.
Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment
Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments...
Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces.
Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments...
Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil>
Duration: 5.762933462s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
5.804687931s trace:
https://jaeger.dev.storj.io/trace/15f47b4548974807
FAIL (2026-01-11 16:02:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload
Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload
Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments...
Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false
Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment...
Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment.
Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment
Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments...
Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces.
Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments...
Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil>
Duration: 5.804718369s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [............X........X......X.X...........X..........X......] duration:
31.574786244s trace:
https://jaeger.dev.storj.io/trace/bb0bc83ebd819e9
FAIL (2026-01-11 16:02:22 UTC)
orbiter warning: context deadline exceeded
Duration: 31.574816441s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
35 out of 60 is failed (threshold=3) [XXX......XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X.] duration:
25.259534451s trace:
https://jaeger.dev.storj.io/trace/2760d69b475b327
FAIL (2026-01-11 16:02:05 UTC)
orbiter warning: context deadline exceeded
Duration: 25.25956523s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.656290808s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 16:02:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 5.656320456s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XX.XXXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXXX] duration:
5.001219986s trace:
https://jaeger.dev.storj.io/trace/2fdf8b804ac97f05
FAIL (2026-01-11 16:02:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001244994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
5.785788252s trace:
https://jaeger.dev.storj.io/trace/15f47b4548974807
FAIL (2026-01-11 16:01:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload
Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload
Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments...
Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false
Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment...
Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment.
Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment
Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments...
Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces.
Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments...
Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil>
Duration: 5.785814402s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.XX.XXXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXXX] duration:
5.002498578s trace:
https://jaeger.dev.storj.io/trace/448adf32ccd2e8e8
FAIL (2026-01-11 16:01:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002522894s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.............X........X......X.X...........X..........X.....] duration:
40.527702049s trace:
https://jaeger.dev.storj.io/trace/bb0bc83ebd819e9
FAIL (2026-01-11 16:01:31 UTC)
orbiter warning: context deadline exceeded
Duration: 40.527732417s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.929903937s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 16:01:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 5.929944505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
35 out of 60 is failed (threshold=3) [.XXX......XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.X] duration:
30.000548076s trace:
https://jaeger.dev.storj.io/trace/2760d69b475b327
FAIL (2026-01-11 16:01:21 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000575608s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.703787508s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 16:01:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 5.703813709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X.XX.XXXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXXX] duration:
5.001854602s trace:
https://jaeger.dev.storj.io/trace/4d953012203c4022
FAIL (2026-01-11 16:00:52 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001889558s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [..XXX......XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X.] duration:
30.001101548s trace:
https://jaeger.dev.storj.io/trace/1d160084793fb8d1
FAIL (2026-01-11 16:00:48 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001131285s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
5.801436887s trace:
https://jaeger.dev.storj.io/trace/15f47b4548974807
FAIL (2026-01-11 16:00:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload
Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload
Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments...
Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false
Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment...
Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment.
Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment
Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments...
Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces.
Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments...
Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil>
Duration: 5.801462466s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..............X........X......X.X...........X..........X....] duration:
25.091928127s trace:
https://jaeger.dev.storj.io/trace/bb0bc83ebd819e9
FAIL (2026-01-11 16:00:32 UTC)
orbiter warning: context deadline exceeded
Duration: 25.092108239s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
5.794104396s trace:
https://jaeger.dev.storj.io/trace/15f47b4548974807
FAIL (2026-01-11 16:00:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload
Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload
Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments...
Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false
Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment...
Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment.
Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment
Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments...
Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces.
Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments...
Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil>
Duration: 5.794128151s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
35 out of 60 is failed (threshold=3) [X..XXX......XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.X] duration:
30.001093748s trace:
https://jaeger.dev.storj.io/trace/1d160084793fb8d1
FAIL (2026-01-11 16:00:15 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001105972s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....XX......................................................] duration:
5.987005781s trace:
https://jaeger.dev.storj.io/trace/15f47b4548974807
FAIL (2026-01-11 16:00:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload
Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload
Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments...
Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false
Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment...
Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment.
Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece...
Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment
Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments...
Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil>
Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces.
Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold
Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments...
Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil>
Duration: 5.987069744s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XX.XX.XXXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XXX] duration:
5.010213673s trace:
https://jaeger.dev.storj.io/trace/4679b000aae19d8
FAIL (2026-01-11 16:00:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.010243109s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.626239994s trace:
https://jaeger.dev.storj.io/trace/db08c1a98355252
FAIL (2026-01-11 15:59:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded
======LOGS======
=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece...
Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment
Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments...
Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil>
Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces.
Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF
Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold
Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments...
Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil>
Duration: 5.626264091s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [.X..XXX......XXXXX...X.XXXXXXXXXXXX..X.X.X.XXXXXX.......XXX.] duration:
26.438893029s trace:
https://jaeger.dev.storj.io/trace/330537a70aae009c
FAIL (2026-01-11 15:59:45 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.438908037s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXX.XX.XXXXX.XXXXX.X.XXX.XXX.XXXXXXXXXXXXXXXXXX..XX.XXXXX.XX] duration:
5.000622932s trace:
https://jaeger.dev.storj.io/trace/4de9247db63e7eda
FAIL (2026-01-11 15:59:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000644452s
core
uplink
us-select-1
us1-placement-20
|