Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...............................X...........................X] duration:
30.001879765s trace:
https://jaeger.dev.storj.io/trace/29e54bf3415d9356
FAIL (2025-12-08 13:58:24 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001910643s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
16.099488417s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:58:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 16.099519506s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X..................................................X.....] duration:
8.548556174s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 13:58:02 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.548578647s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [..XX.XX.X............X.......XXX..XX....X.X..X.X..XX....X...] duration:
17.512650039s trace:
https://jaeger.dev.storj.io/trace/4686fe69f6bea463
FAIL (2025-12-08 13:57:59 UTC)
orbiter warning: context deadline exceeded
Duration: 17.512704523s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001187222s trace:
https://jaeger.dev.storj.io/trace/23c490918abe9456
FAIL (2025-12-08 13:57:48 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001215205s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
18.079851548s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:57:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 18.079886324s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X......X....X...............X....X.X.............X.X] duration:
30.001361851s trace:
https://jaeger.dev.storj.io/trace/795c85bb041d360d
FAIL (2025-12-08 13:57:40 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001388783s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
3.063365589s trace:
https://jaeger.dev.storj.io/trace/279da5a1e905f33b
FAIL (2025-12-08 13:57:39 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 3.063430132s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
4.452780977s trace:
https://jaeger.dev.storj.io/trace/279da5a1e905f33b
FAIL (2025-12-08 13:57:21 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.452822436s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [...XX.XX.X............X.......XXX..XX....X.X..X.X..XX....X..] duration:
19.628813135s trace:
https://jaeger.dev.storj.io/trace/4686fe69f6bea463
FAIL (2025-12-08 13:57:18 UTC)
orbiter warning: context deadline exceeded
Duration: 19.628828323s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
13.950098217s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:57:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 13.950126141s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001486889s trace:
https://jaeger.dev.storj.io/trace/67090793b18a5a21
FAIL (2025-12-08 13:57:02 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00151299s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [....XX.XX.X............X.......XXX..XX....X.X..X.X..XX....X.] duration:
27.258618328s trace:
https://jaeger.dev.storj.io/trace/4686fe69f6bea463
FAIL (2025-12-08 13:56:58 UTC)
orbiter warning: context deadline exceeded
Duration: 27.258633906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [X........X......X....X...............X....X.X.............X.] duration:
22.76035569s trace:
https://jaeger.dev.storj.io/trace/7e7980c7350929ae
FAIL (2025-12-08 13:56:55 UTC)
orbiter warning: context deadline exceeded
Duration: 22.760369937s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X..................................................X....] duration:
7.469527929s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 13:56:53 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.469558669s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.....XX.XX.X............X.......XXX..XX....X.X..X.X..XX....X] duration:
1m0.001603547s trace:
https://jaeger.dev.storj.io/trace/4686fe69f6bea463
FAIL (2025-12-08 13:56:30 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001617453s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X........X......X....X...............X....X.X.............X] duration:
30.000837917s trace:
https://jaeger.dev.storj.io/trace/7e7980c7350929ae
FAIL (2025-12-08 13:56:23 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00085044s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.979217273s trace:
https://jaeger.dev.storj.io/trace/279da5a1e905f33b
FAIL (2025-12-08 13:56:08 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.979251558s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.002307263s trace:
https://jaeger.dev.storj.io/trace/75ad6854ad973faf
FAIL (2025-12-08 13:55:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002329876s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X......X....................................................] duration:
9.909817456s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:55:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 9.909850519s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X..................................................X...] duration:
8.99257373s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 13:55:53 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.992604628s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000534662s trace:
https://jaeger.dev.storj.io/trace/67bca8d91caff166
FAIL (2025-12-08 13:55:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000558437s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [X.....XX.XX.X............X.......XXX..XX....X.X..X.X..XX....] duration:
23.699650347s trace:
https://jaeger.dev.storj.io/trace/47e517302b6bdea5
FAIL (2025-12-08 13:55:30 UTC)
orbiter warning: context deadline exceeded
Duration: 23.699682799s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X......X...................................................] duration:
10.843813049s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:55:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 10.843842976s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..X........X......X....X...............X....X.X.............] duration:
21.877208611s trace:
https://jaeger.dev.storj.io/trace/62ef279b44e8d635
FAIL (2025-12-08 13:55:17 UTC)
orbiter warning: context deadline exceeded
Duration: 21.877297829s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X..................................................X..] duration:
7.255453949s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 13:55:14 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.255483887s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
3.084026044s trace:
https://jaeger.dev.storj.io/trace/279da5a1e905f33b
FAIL (2025-12-08 13:55:05 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 3.084075147s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X......X..................................................] duration:
11.930533083s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:54:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 11.93056335s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...X........X......X....X...............X....X.X............] duration:
26.212238441s trace:
https://jaeger.dev.storj.io/trace/62ef279b44e8d635
FAIL (2025-12-08 13:54:41 UTC)
orbiter warning: context deadline exceeded
Duration: 26.212263901s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X..................................................X.] duration:
7.550405637s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 13:54:34 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.550425015s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001343628s trace:
https://jaeger.dev.storj.io/trace/21ba813ce1d5580d
FAIL (2025-12-08 13:54:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001359078s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.X.....XX.XX.X............X.......XXX..XX....X.X..X.X..XX...] duration:
26.889077177s trace:
https://jaeger.dev.storj.io/trace/47e517302b6bdea5
FAIL (2025-12-08 13:54:29 UTC)
orbiter warning: context deadline exceeded
Duration: 26.889111812s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X..................................................X] duration:
5m19.76716359s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 13:54:26 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5m19.767191142s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
1.992282629s trace:
https://jaeger.dev.storj.io/trace/279da5a1e905f33b
FAIL (2025-12-08 13:53:58 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.992311434s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X......X.................................................] duration:
9.61478485s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:53:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 9.614812092s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001736949s trace:
https://jaeger.dev.storj.io/trace/770f20a65436416e
FAIL (2025-12-08 13:53:32 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001760705s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [..X.....XX.XX.X............X.......XXX..XX....X.X..X.X..XX..] duration:
31.426654022s trace:
https://jaeger.dev.storj.io/trace/47e517302b6bdea5
FAIL (2025-12-08 13:53:24 UTC)
orbiter warning: context deadline exceeded
Duration: 31.426678259s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X......X................................................] duration:
12.745820278s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:53:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 12.745851737s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [....X........X......X....X...............X....X.X...........] duration:
6.434288609s trace:
https://jaeger.dev.storj.io/trace/62ef279b44e8d635
FAIL (2025-12-08 13:53:09 UTC)
orbiter warning: context deadline exceeded
Duration: 6.434330078s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.....X........X......X....X...............X....X.X..........] duration:
29.757332021s trace:
https://jaeger.dev.storj.io/trace/62ef279b44e8d635
FAIL (2025-12-08 13:52:56 UTC)
orbiter warning: context deadline exceeded
Duration: 29.757424205s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001972037s trace:
https://jaeger.dev.storj.io/trace/7019f7dd5b49b1a
FAIL (2025-12-08 13:52:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001992095s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X......X...............................................] duration:
15.621204479s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:52:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 15.621223745s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
1.977164039s trace:
https://jaeger.dev.storj.io/trace/279da5a1e905f33b
FAIL (2025-12-08 13:52:45 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.977191932s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [...X.....XX.XX.X............X.......XXX..XX....X.X..X.X..XX.] duration:
24.35670055s trace:
https://jaeger.dev.storj.io/trace/47e517302b6bdea5
FAIL (2025-12-08 13:52:18 UTC)
orbiter warning: context deadline exceeded
Duration: 24.356713885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration:
5.00102909s trace:
https://jaeger.dev.storj.io/trace/172335de5d3a5f3d
FAIL (2025-12-08 13:52:17 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001049859s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [....X.....XX.XX.X............X.......XXX..XX....X.X..X.X..XX] duration:
1m0.000946815s trace:
https://jaeger.dev.storj.io/trace/47e517302b6bdea5
FAIL (2025-12-08 13:51:53 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000973145s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
3.023192358s trace:
https://jaeger.dev.storj.io/trace/279da5a1e905f33b
FAIL (2025-12-08 13:51:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 3.023219258s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X......X..............................................] duration:
9.897895538s trace:
https://jaeger.dev.storj.io/trace/26d2fdda31045327
FAIL (2025-12-08 13:51:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload
Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload
Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments...
Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false
Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment...
Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment.
Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece...
Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000
Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment
Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments...
Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil>
Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces.
Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF
Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold
Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments...
Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded
Duration: 9.897944253s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [X.....X........X......X....X...............X....X.X.........] duration:
8.609379247s trace:
https://jaeger.dev.storj.io/trace/62ef279b44e8d635
FAIL (2025-12-08 13:51:48 UTC)
orbiter warning: context deadline exceeded
Duration: 8.609398042s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X.....X........X......X....X...............X....X.X........] duration:
22.361242602s trace:
https://jaeger.dev.storj.io/trace/62ef279b44e8d635
FAIL (2025-12-08 13:51:40 UTC)
orbiter warning: context deadline exceeded
Duration: 22.361260655s
edge
gateway
us-select-1
|