Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 863749 reports (Page 11720 of 17275)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X...X.X.....X........X......X....X.........] duration: 13.811559823s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:36:18 UTC) orbiter warning: context deadline exceeded Duration: 13.811574061s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......X...........................] duration: 8.779469541s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:36:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.779502013s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [...X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.] duration: 3.472242116s trace: https://jaeger.dev.storj.io/trace/469dc8bed1f860fa FAIL (2025-12-08 13:36:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472408281s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................X...X.X.....X........X......X....X........] duration: 24.867176206s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:36:03 UTC) orbiter warning: context deadline exceeded Duration: 24.867200182s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.124028237s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:35:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.124059779s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......X..........................] duration: 9.350632298s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:35:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.350663258s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 475.49043ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:35:37 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.52252ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [....X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration: 5.001201311s trace: https://jaeger.dev.storj.io/trace/469dc8bed1f860fa FAIL (2025-12-08 13:35:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001223282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 7.993991255s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:35:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.994019869s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 467.782201ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:35:20 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.805935ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.....X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration: 5.002138226s trace: https://jaeger.dev.storj.io/trace/3bfb68edee050e93 FAIL (2025-12-08 13:35:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002165137s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X...X.X.....X........X......X....X.......] duration: 18.989877921s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:35:06 UTC) orbiter warning: context deadline exceeded Duration: 18.989937124s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [......X.........X.....X.....XX.XX.X............X.......XXX..] duration: 18.521590581s trace: https://jaeger.dev.storj.io/trace/32e5cc8650207f35 FAIL (2025-12-08 13:34:58 UTC) orbiter warning: context deadline exceeded Duration: 18.521619857s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 676.734478ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:34:49 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 676.756521ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [......X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 5.001724114s trace: https://jaeger.dev.storj.io/trace/4827909885de1cc5 FAIL (2025-12-08 13:34:37 UTC) orbiter warning: context deadline exceeded Duration: 5.001742158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......X.........................] duration: 7.813272814s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:34:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.813306709s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 682.105137ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:34:25 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 682.130074ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.......X.........X.....X.....XX.XX.X............X.......XXX.] duration: 24.384293851s trace: https://jaeger.dev.storj.io/trace/32e5cc8650207f35 FAIL (2025-12-08 13:34:24 UTC) orbiter warning: context deadline exceeded Duration: 24.384317596s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 476.410145ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:34:17 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.445854ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [X......X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 5.001702214s trace: https://jaeger.dev.storj.io/trace/532b1b0b08094b58 FAIL (2025-12-08 13:34:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001719897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......X........................] duration: 20.610346838s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:34:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 20.610374963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.454874875s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:34:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.454902066s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 685.4367ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:33:56 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 685.499099ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X...X.X.....X........X......X....X......] duration: 20.034927125s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:33:52 UTC) orbiter warning: context deadline exceeded Duration: 20.034941783s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [........X.........X.....X.....XX.XX.X............X.......XXX] duration: 1m0.001100097s trace: https://jaeger.dev.storj.io/trace/32e5cc8650207f35 FAIL (2025-12-08 13:33:50 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001111859s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 472.954925ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:33:46 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 472.980263ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.945133494s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:33:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.945169723s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.X......X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.001444971s trace: https://jaeger.dev.storj.io/trace/6830fe44242fc2ad FAIL (2025-12-08 13:33:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001469767s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X...X.X.....X........X......X....X.....] duration: 25.737059598s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:33:32 UTC) orbiter warning: context deadline exceeded Duration: 25.73708666s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 994.449678ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:33:18 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 994.543849ms core satellite us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 471.000955ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:33:11 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 471.025833ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [..X......X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 5.00157178s trace: https://jaeger.dev.storj.io/trace/3c21c39847146673 FAIL (2025-12-08 13:32:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001596548s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.162763187s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:32:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 2.162804846s core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.........X.........X.....X.....XX.XX.X............X.......XX] duration: 1m0.001479408s trace: https://jaeger.dev.storj.io/trace/29188267e07eaf65 FAIL (2025-12-08 13:32:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001503664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......X.......................] duration: 8.033684775s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:32:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.033706518s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X...X.X.....X........X......X....X....] duration: 24.152503707s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:32:40 UTC) orbiter warning: context deadline exceeded Duration: 24.152519657s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.572904887s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:32:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.572930927s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [X..X......X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 5.001672775s trace: https://jaeger.dev.storj.io/trace/1c683de6d82d5616 FAIL (2025-12-08 13:32:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001689126s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 705.942597ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:32:24 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 705.970119ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......X......................] duration: 8.609210468s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:32:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.609246046s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 476.770722ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:31:59 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.796681ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......X.....................] duration: 8.031030486s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:31:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.031060061s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..........X.........X.....X.....XX.XX.X............X.......X] duration: 1m0.001259864s trace: https://jaeger.dev.storj.io/trace/782567c9636881a9 FAIL (2025-12-08 13:31:43 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001283631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.X..X......X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.00065696s trace: https://jaeger.dev.storj.io/trace/5a03f109850b6c79 FAIL (2025-12-08 13:31:37 UTC) orbiter warning: context deadline exceeded Duration: 5.000679402s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.302456869s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:31:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.30248384s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................X...X.X.....X........X......X....X...] duration: 15.336264169s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:31:28 UTC) orbiter warning: context deadline exceeded Duration: 15.336289277s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.056154405s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:31:11 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.056196156s core satellite us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 470.900093ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:30:59 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.933346ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X...X.X.....X........X......X....X..] duration: 24.475968163s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:30:57 UTC) orbiter warning: context deadline exceeded Duration: 24.475996215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X......X....................] duration: 8.470132994s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:30:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.47016342s core uplink us-select-1

1
...
11718
11719
11720
11721
11722
...
17275